Systems and methods to optimize the load of multipath data transportation

ABSTRACT

A control unit of a multipath data transportation system that optimizes the load of the multiple communication paths of this system when the system transmits a data segment over these paths in parallel with forward error correction. The control unit determines an optimized number of packets to send over each path based on a prediction of quality for each path. The transmitted packets include systematic packets and coded packets.

RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/714,303, filed Sep. 25, 2017, which is related to U.S. patentapplication Ser. No. 15/405,874, filed Jan. 13, 2017, and entitled“SYSTEMS AND METHODS TO GENERATE COPIES OF DATA FOR TRANSMISSION OVERMULTIPLE COMMUNICATION CHANNELS,” and is also related to U.S. patentapplication Ser. No. 15/411,097, filed Jan. 20, 2017, and entitled“SYSTEMS AND METHODS TO OPTIMIZE PARTITIONING OF A DATA SEGMENT INTODATA PACKETS FOR CHANNEL ENCODING,” the disclosures of which are herebyincorporated herein by reference.

FIELD OF THE TECHNOLOGY

At least some embodiments disclosed herein relate to encoding data forreliable and efficient transmission over a communication system thatdoes not have perfect fidelity, such as a communication system havingparallel channels for communications made using forward error correctiontechniques.

BACKGROUND

Forward Error Correction (FEC) is a coding technique typically used forcorrecting errors in data transmission over communication channels.Using FEC a sender provides redundant data in the stream of communicateddata to enable the recipient to detect errors in the received dataand/or recover a copy of the original data from the correctly receiveddata without requiring a reverse communication from the recipient to thesender to request retransmission of missing data or received data thathas error.

FEC adds redundant data by transforming the original information intochannel-encoded outputs using an algorithm. The original information mayor may not appear literally in the encoded output. Outputs that includethe unmodified input are systematic, while outputs that do not includethe unmodified input are non-systematic. A recipient decodes thenon-systematic outputs to reassemble the original information from thereceived outputs. Rateless erasure codes can potentially generate alimitless sequence of encoding symbols from a given set of originalsymbols such that the original symbols can be recovered from a subset ofthe encoding symbols of size equal to or larger than the number oforiginal symbols. The rateless FEC does not have a fixed code rate.

U.S. Pat. No. 9,413,494, entitled “FEC-based Reliable Transport ControlProtocols for Multipath Streaming”, discloses the transmission offorward-error corrected data, from a server device to a client device,via a plurality of parallel network paths. The client device reports, tothe server device, the losses of the data in the network paths for theretransmission of the data lost in the network paths.

U.S. Pat. No. 7,249,291, entitled “System and Method for ReliablyCommunicating the Content of a Live Data Stream”, discloses a method totransmit a live data stream to a recipient using multiple channels,where a forward error correction algorithm is applied to generateFEC-encoded blocks.

U.S. Pat. No. 9,236,885, entitled “Systematic Encoding and Decoding ofChain Reaction Codes”, discloses that in certain applications it may bepreferable to transmit the source symbols first, and then to continuetransmission by sending output symbols. Such a coding system wasreferred to as a systematic coding system.

U.S. Pat. No. 9,015,564, entitled “Content Delivery System withAllocation of Source Data and Repair Data among HTTP Servers”, disclosesa source server storing content in a source form, a broadcast severgenerating and storing repair symbols, and repair servers requestingbroadcast of repair data when byte-range requests from multiplereceivers overlap.

U.S. Pat. App. Pub. No. 2010/0094955, entitled “Methods and Systems forusing a Distributed Storage to its Maximum Bandwidth”, discloses amethod to generate multiple coded fractional copies of a data segment,which are stored in multiple storage devices. A subset of the codedcopies will be fully downloaded for data recovery, and in case ofrecovery failure, more coded copies will be fully downloaded.

U.S. Pat. No. 8,078,746, entitled “Distribution Method, PreferablyApplied in a Streaming System”, discloses a method to generate, at peerreceivers of a network, FEC-coded feed packets of the portion of thebroadcast data they have received, so that all peer receivers candownload sufficient feed packets from any other peer receivers torecover the entire broadcast data.

U.S. Pat. No. 8,996,946, entitled “Application of Fountain Forward ErrorCorrection Codes in Multi-link Multi-path Mobile Networks”, discloses amethod to generate different on-the-fly FEC encoded packets of a datasegment and send these packets through different unreliable paths thatare coupled with a receiver until the receiver has acknowledged or thetime is out.

U.S. Pat. App. Pub. No. 2014/0269289, entitled “Method and Apparatus forImproving Communication Performance through Network Coding”, discloses amethod to generate different on-the-fly network coding encoded packetsof a data segment and send these packets through different unreliablepaths that are coupled with a receiver. The receiver may sendintermediate feedback to the transmitter to manage retransmissions. Incase the data segment cannot be fully recovered, partial recovery willbe attempted.

U.S. Pat. No. 9,455,750, entitled “Source Block Size Selection”,discloses techniques to select a source block size to reduce receptionoverhead in a communication system that partitions a media stream intosource blocks of the selected size, divides each of the source blockinto source symbols that are encoded to generate repair symbols fortransmission over a network.

The entire disclosures of the above identified patents are herebyincorporated herein by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments are illustrated by way of example and not limitation inthe figures of the accompanying drawings in which like referencesindicate similar elements.

FIG. 1 shows a technique to transmit a data segment via FEC overmultiple parallel channels according to one embodiment.

FIG. 2 shows a technique to transmit a data segment via FEC overmultiple parallel channels using an initial systematic phase accordingto one embodiment.

FIG. 3 shows a technique to transmit a data segment via FEC overmultiple parallel channels using encoded packets that are linearlyindependent of their base sets according to one embodiment.

FIG. 4 shows scheduling of systematic packets over multiple parallelchannels according to one embodiment.

FIG. 5 shows the identification of the composition of a channel-encodedpacket according to one embodiment.

FIG. 6 shows a system to transmit data packets using multiple channelsaccording to one embodiment.

FIG. 7 shows a method to generate a channel-encoded packet according toone embodiment.

FIG. 8 shows a method to transmit a data segment according to oneembodiment.

FIG. 9 illustrates an overhead cost in data communication fortransmitting a data segment by dividing the data segment into aplurality of data packets for forward error control.

FIG. 10 illustrates the data structure of a communication packet of oneembodiment.

FIG. 11 illustrates various cost considerations in the transmission of adata segment using a forward error correction technique.

FIG. 12 shows a method to transmit a data segment according to oneembodiment.

FIG. 13 shows a data processing system on which the methods of thepresent disclosure can be implemented.

FIG. 14 shows a multipath data transportation system having a controlunit implemented at a server side according to one embodiment.

FIG. 15 shows a flow chart of the control unit of the multipath datatransportation system according to one embodiment.

FIG. 16 illustrates an ideal simultaneous download completion ofsystematic packets without packet loss according to one embodiment.

FIG. 17 illustrates an ideal case of simultaneous download completionwith packet loss, no redundancy after decoding according to oneembodiment.

FIG. 18 illustrates asynchronous download times, in which Path-1 is notcontributing between t4 and t5 according to one embodiment.

FIG. 19 illustrates a receiver that receives redundant packets from bothpaths after it has decoded according to one embodiment.

FIG. 20 shows a relocation algorithm when there is a transmission windowtime that limits {K(n)} according to one embodiment.

FIG. 21 shows a complete {K(n)} optimization algorithm according to oneembodiment.

FIG. 22 shows an {S(n)} optimization algorithm that maximizes the numberof received systematic packets according to one embodiment.

FIG. 23 shows a complete multipath load balancing algorithm according toone embodiment.

FIG. 24 illustrates an exemplary relation between transmission rate andreceiver observed PLR when BW=5 Mbps and PLR=5% according to oneembodiment.

DETAILED DESCRIPTION

The following description and drawings are illustrative and are not tobe construed as limiting. Numerous specific details are described toprovide a thorough understanding. However, in certain instances, wellknown or conventional details are not described in order to avoidobscuring the description. References to one or an embodiment in thepresent disclosure are not necessarily references to the sameembodiment; and, such references mean at least one.

At least some embodiments disclosed herein provide solutions to improvethe efficiency, reliability, and overall speed of data transmission viamultiple communication channels, in a system as illustrated in FIG. 1.The data transmission technique uses a rateless FEC technique, whichpartitions the original data segment into original packets of anequal-length and generates channel-coded packets from linearcombinations of various subsets of the original packets.

FIG. 1 shows a technique to transmit a data segment via FEC overmultiple parallel channels according to one embodiment.

In FIG. 1, a data segment (111) is transmitted via a plurality ofparallel communication channels (105, 107, . . . , 109). The use of theparallel communication channels (105, 107, . . . , 109) generallyincreases the overall speed of the transmission of the data segment(111).

However, the actual performances of the individual communicationchannels (105, 107, . . . , 109) may vary at the time of transmission.Packets transmitted in one or more of the channels (e.g., 105, 107, . .. , or 109) may have longer, random delays than other packetstransmitted via others channels (e.g., 105, 107, . . . , or 109). As aresult, the sequence of the combined set of packets received in arecipient device from the channels (105, 107, . . . , 109) is randomizedat the time of transmission.

Thus, it is a challenge to schedule the packets for the transmission inthe communication channels (105, 107, . . . , 109) with reducedredundancy data and a reduced time period during which a sufficientnumber of useful packets are received via the parallel communicationchannels (105, 107, . . . , 109) for the reconstruction of the recovereddata segment (113) that is the same as the original data segment (111).

In FIG. 1, the data segment (111) is partitioned (101), or divided, intoa plurality of packets (121, 123, 125, . . . , 127) of the same length.Dummy bytes may be padded so that the packets (121, 123, 125, . . . ,127) have the same length and thus allow the use of a forward errorcorrection technique. Using a forward error correction technique, thesystem channel encodes (103) the original packets (121, 123, 125, . . ., 127) to generate different channel-encoded copies (115. 117, . . . ,119) of the data segment (111) that carry redundant information in thecopies (115, 117, . . . , 119) for the respective channels (105, 107, .. . , 119).

For example, in FIG. 1, a typical channel-encoded packet (e.g., 131,133, 135, . . . , 139; 141, 143, 145, . . . , 149; . . . ; 151, 153,157, . . . , or 159) generated using a typical forward error correctiontechnique is a predetermined function of a random subset of originalpackets (121, 123, 125, . . . , 127). The predetermined functiontypically generates a linear combination of the subset of the originalpackets (121, 123, 125, . . . , 127) (e.g., through an exclusive or(XOR) operation) as a channel-encoded packet. When a sufficient numberof channel-encoded packets are received via the channels (105, 107, . .. , 109), the received set of channel-encoded packets can be decoded viathe respective forward error correction technique to provide theoriginal set of packets (121, 123, 125, . . . , 127) and thus therecovered data segments (113).

In FIG. 1, the individual channels (e.g., 105, 107, . . . , or 109) maybe reliable in the quality of data transmission (e.g., using a reliableprotocol). When there is a missing packet or a packet having an error isreceived, the retransmission of the missing packet or packet having anerror can be requested through any channels (e.g., 105, 107, . . . , or109) using an appropriate protocol. However, the speed performances ofchannels (e.g., 105, 107, . . . , or 109) may be not be consistent,which incurs delays and duplications to the retransmissions. Embodimentsdisclosed herein provide solutions to improve the performance of thesystem by reducing duplicated (and thus useless) data and reducing thetime period for the transmission of a sufficient set of useful packetsfor the recovery of the data segment (113).

In some instances, it is desirable to use separate transmitters to sendthe copies (115, 117, . . . , 119) through the channels (105, 107, . . ., 109) independently from each other. The transmitters may not havecommunications with each other to coordinate their transmissions of thecopies (115, 117, . . . , 119). The system of FIG. 1 is configured suchthat when a recipient device receives a combined set of sufficientpackets from channels (105, 107, . . . , 109), the recipient device cangenerate, from the combined set of received packets, the recovered datasegment (113) that is the same as the original data segment (111),without having to wait for the completion transmission of the copies(115, 117, . . . , 119). The recipient device may simply terminate itsreceiving operations before the complete transmission of the copies(115, 117, . . . , 119) and/or request the termination of thetransmission operations.

For example, when a combined set of sufficient packets is received forthe recovered data segment (113), the recipient device provides anindication to the transmitting system to terminate the transmission ofthe remaining packets of the copies (115, 117, . . . , 119) in thechannels (105, 107, . . . , 109), such that the network resources can beused for other transmission tasks.

For example, the recipient device may use the channels (115, 117, . . ., 119) to request the respective transmitters coupled to the channels(115, 117, . . . , 119) to terminate their transmission of their copies(e.g., 115, 117, . . . , 119). Alternatively, the recipient device maynotify a centralized server, using one of the channels (105, 107, . . ., 109) or a separate communication channel, which server then furthernotifies the respective transmitters coupled to the respective channels(115, 117, . . . , 119) to terminate their transmissions.

The system of FIG. 1 can be further improved by distributing theoriginal packets as initial packets (or leading packets) in the copies(115, 117, . . . , 119) that are transmitted prior to the transmissionof channel-encoded packets (or non-systematic) of the respective copies(115, 117, . . . , 119), as illustrated in FIG. 2.

Alternatively or in combination, the system of FIG. 1 can be furtherimproved by rejecting, in the random selection of the original packetsfor channel-encoding using the forward error correction technique,channel-encoded packets that do not, or are not likely to, contribute tothe recovered data segment (113), in view of a base set of packets thathave been, or likely, received by the recipient device before thetransmission of such packets. Thus, data redundancy is reduced withoutimpacting the performance of forward error correction, as illustrated inFIG. 3.

In general, the techniques of FIG. 2 and FIG. 3 can be combined.

Further, fora given number of the original packets (121, 123, 125, . . ., 127) and a given number of the channels (e.g., 105, 107, . . . , 109),the system can predetermine the composition of the channel-encodedpackets (131, 133, 135, . . . , 139; 141, 143, 145, . . . , 149; . . . ;151, 153, 157, . . . , 159). The composition of a channel-encoded packet(131, 133, 135, . . . , 139; 141, 143, 145, . . . , 149; . . . ; 151,153, 157, . . . , or 159) identifies the identity/identities of thesubset of the original packets (121, 123, 125, . . . , 127) used togenerate the channel-encoded packet. For example, each of the originalpackets (121, 123, 125, . . . , 127) can be identified based on itssequence/position number in the sequence of the original packets (121,123, 125, . . . , 127) (or another type of identification symbol)without its actual data of the original packet. The compositions of thechannel-encoded packets (131, 133, 135, . . . , 139; 141, 143, 145, . .. , 149; . . . ; 151, 153, 157, . . . , 159) for the encoded copies(115, 117, . . . , 119) can be stored without the actual data of theoriginal packets. When the actual data of the original packets (121,123, 125, . . . , 127) is available, the compositions of thechannel-encoded packets (131, 133, 135, . . . , 139; 141, 143, 145, . .. , 149; . . . ; 151, 153, 157, . . . , 159) are retrieved to combinethe respective original packets (121, 123, 125, . . . , 127) identifiedby the compositions to generate the respective channel-encoded packets(131, 133, 135, . . . , 139; 141, 143, 145, . . . , 149; . . . ; 151,153, 157, . . . , 159).

For example, at the time of the transmission, a transmitter receives theset of original packets (121, 123, 125, . . . , 127) and the sequence ofcompositions for the scheduled sequence of channel-encoded packets(e.g., 131, 133, 135, . . . , 139; 141, 143, 145, . . . , 149; . . . ;or 151, 153, 157, . . . , 159). The transmitter combines the originalpackets (121, 123, 125, . . . , 127) according to the compositions togenerate the respective channel-encoded packets just before transmittingthe generated packets for transmission over the respective channel(e.g., 105, 107, . . . , or 109). Thus, when the recipient devicesignals for the early termination of the transmission, the remainingchannel-encoded packets do not have to be actually generated and/orstored.

FIG. 2 shows a technique to transmit a data segment via FEC overmultiple parallel channels using an initial systematic phase accordingto one embodiment. For example, the technique of FIG. 2 can be used in asystem illustrated in FIG. 1.

In FIG. 2, each copy (e.g., 115, 117, . . . , or 119) scheduled to betransmitted via a respective channel (e.g., 105, 107, . . . , 109) has asequence of scheduled packets (121, 127, 135, . . . , 139; 123, 143,145, . . . , 149; . . . ; or 125, 153, 157, . . . , 159). The copies(115, 117, . . . , 119) have a set of initial packets (121, 123, . . . ,125, 127) that correspond to the set of original packets (121, 123, 125,. . . , 127).

In each of the channels (105, 107, . . . , 109), the initial packets ofa respective copy (e.g., 115, 117, . . . , 119) are transmitted beforeother packets of the respective copy (e.g., 115, 117, . . . , 119). Forexample, the initial packets (121, 127) of Copy X (115) are transmittedbefore the transmission of packets (135, . . . , 139) of Copy X (115);the initial packet (123) of Copy Y (117) is transmitted before thetransmission of packets (143, 145, . . . , 149) of Copy Y (117); and theinitial packet (125) of Copy Z (119) is transmitted before thetransmission of packets (153, 155, . . . , 159) of Copy Z (119).

The initial packets (121, 123, . . . , 125, 127) aregenerated/identified by distributing the set of original packets (121,123, 125, . . . , 127) to the copies (115, 117, . . . , 119).

For example, when the channels (105, 107, . . . , 109) have the samenominal/average/expected transmission speed, the original packets (121,123, 125, . . . , 127) can be distributed to the copies (115, 117, . . ., 119) substantially evenly. For example, the original packets (121,123, 125, . . . , 127) can be distributed one at a time to the copiesaccording to a round-robin scheme until all of the original packets(121, 123, 125, . . . , 127) are distributed to the copies (115, 117, .. . , 119). Alternatively, the system computes the average number oforiginal packets (e.g., by dividing the number of packets by the numberof channels and taking the integer part of the result), assigns theaverage number of original packets to each copy, and randomly distributethe remaining, unassigned packets one at a time to the copies (115, 117,. . . , 119) until all of the original packets (121, 123, 125, . . . ,127) are assigned.

In instances where the channels (105, 107, . . . , 109) havesubstantially different nominal/average/expected transmission speeds,the original packets (121, 123, 125, . . . , 127) can be distributed tothe copies (115, 117, . . . , 119) in a ratio approximately equal to theratio of their nominal/average/expected transmission speeds.

Once all original packets have been distributed as the initial packetsof the copies (e.g., 115, 117, . . . , 119), channel-encoded packets(e.g., 135, 143, 145, . . . , 149, 153, 157, . . . , 159) are generatedand added to the copies (e.g., 115, 117, . . . , 119). Eachchannel-encoded packet (e.g., 135) of a given copy (e.g., 115) can begenerated from a linear combination of a random selected subset of allthe original packets (e.g., 121, 123, 125, . . . , 127).

Preferably, each channel-encoded packet (e.g., 135) of a given copy(e.g., 115) can be generated from a linear combination of a randomselected subset of the original packets (e.g., 123, 125) carried byother copies (e.g., 117, . . . , 119) another than the given copy (e.g.,115). In other words, the set of original packets (e.g., 121, 127)carried by the given copy (e.g., 115) is not used for the generation ofthe channel-encoded packets (e.g., 135, . . . , 139) of this given copy(e.g., 115). This is because by the time a channel-encoded packet (e.g.,135) of this given copy (e.g., 115) is received, the recipient devicewill already have received all the original packets (e.g., 121, 127)carried by this given copy (e.g., 115). Excluding these original packetswill reduce both computational complexity and the probability ofgenerating useless channel-encoded packets.

FIG. 3 shows a technique to transmit a data segment via FEC overmultiple parallel channels using channel-encoded packets that arelinearly independent of their base sets according to one embodiment. Forexample, the technique of FIG. 3 can be used in a system illustrated inFIGS. 1 and/or 2.

In FIG. 3, the transmission of data packets within each channel (105,107, . . . , 109) is reliable. Thus, at the time of the transmission ofa subsequent packet (e.g., 135) in the channel (e.g., 105), therecipient device has correctly received the data packets (e.g., 131,133) that have been transmitted in the same channel (e.g., 105) beforethe subsequent packet (e.g., 135). Thus, a candidate for the subsequentchannel-encoded packet (e.g., 135) that is linearly dependent of thereceived data packets (e.g., 131, 133) is useless and can be rejected.

Further, by the time of the transmission of the subsequent packet (e.g.,135) in the channel (e.g., 105), it is very likely that the recipientdevice has received the initial packets (e.g., 141, . . . , 151)transmitted in other channels (e.g., 107, . . . , 109). The subsequentpacket (e.g., 135) can be considered to have a base set (161) of packets(131, 133, 141, . . . , 151) that include all the packets (e.g., 131,133) in the same channel (e.g., 105) that are scheduled to betransmitted before the subsequent packet (e.g., 135) and the initialpackets (e.g., 141, . . . , 151) of the other channels (e.g., 107, . . ., 109) that are very likely to have reached the recipient device viaother channels (e.g., 107, . . . , 109) before the transmission of thesubsequent packet (e.g., 135). A candidate for the subsequent packet(e.g., 135) that is linearly dependent of the base set is most likelyuseless and thus can be rejected.

To ensure/improve the usefulness of the channel-encoded packet (135),the system of FIG. 3 is configured to identify the subsequentchannel-encoded packet (135) such that it is linearly independent of thebase set (161) of packets (131, 133, 141, . . . , 151), where the baseset (161) includes all the packets (e.g., 131, 133) that are scheduledin the copy (115) before the subsequent channel-encoded packet (135),and the certain initial packets (e.g., 141, . . . , 151) of the othercopies (117, . . . , 119).

In determining the initial packets (e.g., 141, . . . , 151) of the othercopies (117, . . . , 119) for the base set (161) for the subsequentpacket (135), it is assumed that the base set (161) is insufficient togenerate the recovered data segment (113), which triggers the need totransmit the subsequent packet (135) in the channel (105). Thus, thebase set (161) is identified to include less packets than the entire setof the original packets (121, 123, 125, . . . , 127).

Further, the other channels (e.g., 107, . . . , 109) are assumed to havetransmitted their initial packets according to their ratio of thenominal/average/expected transmission speeds. For example, when theother channels (e.g., 107, . . . , 109) have an equalnominal/average/expected transmission speed, the same number of initialpackets are selected from the other copies (e.g., 117, . . . , 119) forthe base set (161) of the subsequent packet (135).

For example, the initial packets (e.g., 141, . . . , 151) of the othercopies (117, . . . , 119) can be selected for the base set (161) of thesubsequent packet (135) of the copy (115) in the follow manner. Thesystem determines a remaining number of packets to be transmitted in theother copies (117, . . . , 119) by reducing the number of originalpackets (121, 123, 125, . . . , 127) by the number of initial packets inthe copy (115) up to and including the subsequent packet (135). Thesystem then distributes the remaining number of packets to the othercopies (117, . . . , 119) (e.g., according to their ratio ofnominal/average/expected transmission speeds). The numbers of packetsdistributed to the other copies (117, . . . , 119) identifies thecorresponding numbers of initial packets (e.g., 141, . . . , 151) of theother copies (117, . . . , 119) that are included in the base set (161)for the subsequent packet (135) of the copy (115).

For example, there are K original packets (121, 123, 125, . . . , 127);the subsequent packet (135) is the k′th packet in the copy (115); andthe system uses M channels (105, 107, . . . , 109) to transmit the Mcopies (115, 117, . . . , 119) respectively in parallel. The remainingnumber of packets to be distributed to the other copies (117, . . . ,119) is (K−k). When the other channels (107, . . . , 109) have the samenominal/average/expected transmission speed, the other copies (e.g.,117, . . . , 119) have an average of (K−k)/(M−1) initial packets oftheir respective copies (117, . . . , 119) that are selected as the baseset (161) for the subsequent packet (135). When (K−k)/(M−1) is not aninteger, and d is the integer that is closest to (K−k)/(M−1) but nolarger than (K−k)/(M−1), the base set (161) may be selected to includethe initial d packets of each of the other copies (e.g., 117, . . . ,119). Optionally, the base set (161) may further include an additionalinitial packet from (K−k)−d*(M−1) randomly selected copies of the othercopies (e.g., 117, . . . , 119).

Further, when the channels (105, 107, . . . , 109) have the samenominal/average/expected transmission speed, the number of initialpackets selected from each other copies (e.g., 117, . . . , 119) is notmore than k−1, since at the time of the transmission of the k′th packetin channel (105), no more than k−1 packets of each other copies (e.g.,117, . . . , 119) are expected to have been transmitted through eachother channels (107, . . . , 109).

The compositions of the channel-encoded packets can be used to identifywhether or not a candidate for the subsequent packet (135) is linearlydependent of the base set (161). The composition of a channel-encodedpacket (e.g., 135) identifies a subset of the original packets (121,123, 125, . . . , 127) that are used in the generation of thechannel-encoded packet (e.g., 135). More specifically, the compositionof the channel-encoded packet (e.g., 135) identifies which of theoriginal packets (121, 123, 125, . . . , 127) are used in the generationof the channel-encoded packet (e.g., 135) and which are not. Forexample, the composition of a channel-encoded packet can be representedby a vector of symbols, where the i′th symbol of vector is either zerowhen the i′th original packet is not used in the generation of thechannel-encoded packet, or one when the i′th original packet is used inthe generation of the channel-encoded packet.

The subsequent packet (135) can be selected such that the compositionvector of the subsequent packet (135) is linearly independent of thecomposition vectors of the packets (131, 133, 141, . . . , 151) in thebase set (161). For example, when a forward error correction techniqueis used to select a random subset of the original packets (121, 123,125, . . . , 127) for the generation of the subsequent packet (135), thesystem checks whether the random subset has a composition that islinearly independent of the compositions of the packets in the base set(161); if so, the random subset is accepted as the composition of thesubsequent packet (135); otherwise, the random subset is rejected, andanother random subset is selected.

Since each subsequent packet (e.g., 135) scheduled in a copy (e.g., 115)is at least linearly independent of the packets scheduled before thesubsequent packet in the same copy (e.g., 115), the copy (e.g., 115)requires at most the same count of packets (e.g., 131, 133, 135, . . . ,139) as the original packets (121, 123, 125, . . . , 127).

FIG. 4 shows scheduling of systematic packets over multiple parallelchannels according to one embodiment. For example, the schedulingtechnique of FIG. 4 can be used in the system of FIG. 2.

In FIG. 4, the initial packets (121, 123, . . . , 125, 127) of thecopies (115, 117, . . . , 119) are generated by round-robin allocation(163) of the original set of packets (121, 123, . . . , 125, 127) to thecopies (115, 117, . . . , 119). As discussed above, the round-robinallocation (163) can be performed according to the ratio of thenominal/average/expected transmission speeds of channels used totransmit the copies (115, 117, . . . , 119) in parallel.

The round-robin allocation (163) can be performed without the actualdata of the original packets (121, 123, . . . , 125, 127). Thecompositions of the initial systematic packets (121, 123, . . . , 125,127) identify the respective original packets (121, 123, . . . , 125,127) that are literally copied into the respective initial systematicpackets (121, 123, . . . , 125, 127).

Subsequent channel-encoded packets for the copies (115, 117, . . . ,119) can be generated using any forward error correction technique ingeneral and more preferably, in a way as illustrated in FIG. 3 or FIG.5.

FIG. 5 shows the identification of the composition (166) of achannel-encoded packet (143) according to one embodiment. For example,the scheduling technique of FIG. 5 can be used in combination thetechniques of FIG. 3 or FIG. 4. Although FIG. 5 illustrates an examplewhere a set of the initial packets of the copies (115, 117, . . . , 119)are original packets, the method of FIG. 5 can also be applied to theuse of channel-encoded initial packets in the copies (115, 117, . . . ,119).

In FIG. 5, a base set (165) is identified for the packet (143) scheduledto be transmitted after the packet (123) in the copy (117). The base set(165) is insufficient to reconstruct the entire set of the originalpackets (121, 123, . . . , 125, 127); and the base set (165) includespackets that are expected (or most likely) to have been received in arecipient device at the time the packet (143) is to be transmitted ifthe copies (115, 117, . . . , 119) are transmitted in parallel usingmultiple channels (e.g., 105, 107, . . . , 109). Examples of methods toidentify the base set (165) are discussed above in connection with FIG.3.

In FIG. 5, the compositions of the packets in the base set (165) areused to identify the composition (166) of the channel-encoded packet(143). Specifically, the composition (166) is generated by randomlyselecting (167) a subset from the original packets (121, 123, . . . ,125, 127) (e.g., according to a distribution function of a forward errorcorrection technique). In FIG. 5, solid boxed packets (e.g., 121, . . ., 125) in the composition (166) are illustrated as the packets of thesubset that is selected for the generation of the channel-encoded packet(143); and the dash boxed packets (e.g., 123, . . . , 127) in thecomposition (166) are illustrated as the packets not selected for thegeneration of the channel-encoded packet (143). In FIG. 5, the randomselection of the subset is repeated (e.g., according to thepredetermined distribution function) until the composition vector of theselected subset is linearly independent of the composition vectors ofthe packets (e.g., 121, 123, . . . , 125) in the base set (165). Theselected composition (166) of the channel-encoded packet (143) ensuresthat the packet (143) is useful, and thus not redundant, when it is usedin combination with the packets (121, 123, . . . , 125) in base set(165) during channel decoding. Since at least some useless packets arepre-filtered during the selection of the compositions of the subsequentchannel-encoded packets, the technique improved the efficiency in theuse of the network channels during the transmission of thechannel-encoded copies (115, 117, . . . , 119).

Further, to reduce computational complexity the composition (166) can beselected such that the packets (e.g., 121, . . . , 125) selected in thecomposition (166) to compute the channel-encoded packet (143) of thecopy (117) do not include the original packets (e.g., 123) that arescheduled to be transmitted in the copy (117) before the channel-encodedpacket (143) and/or do not include the original packets (121, 123, . . ., 125) in the base set (165). For example, the random selection ofpackets for the composition (166) may be limit to the subset of originalpackets that do not include the leading original packets (e.g., 123) ofthe copy (117) and/or the original packets (121, 123, . . . , 125) inthe base set (165) to reduce both computational complexity inchannel-encoding and the probability of generating a composition that islinearly dependent of the base set (165).

The composition (166) can be stored as an intermediate result such thatat a later time when the actual packet (143) is to be used (e.g., fortransmission over a communication channel), the composition (166) can beused to select original packets that are combined (169) (e.g., viaexclusive or (XOR) or another linear combination function) to generatethe content of the channel-encoded packet (143).

Since the scheduling of the subsequent packet (143) does not require theactual data/content of the original packets (121, 123, . . . , 125,127), the selection (167) of original packets (121, 123, . . . , 125,127) for further subsequent packets can be performed in a similar waybased on the stored compositions of the packets that have alreadyscheduled in the copies (115, 117, . . . , 119). Thus, the compositionsof all of the packets in the copies (115, 117, . . . , 119) can bepre-computed and stored for the predetermined number of original packets(121, 123, . . . , 125, 127) without the actual packet data/content.This allows the scheduling of the packets before transmission and beforethe content of the packets (121, 123, . . . , 125, 127) becomesavailable. At the time of transmission, a transmitter retrieves thecompositions of the packets scheduled in the respective copy (e.g., 117)and combines the original packets according to the compositions (e.g.,166) to generate the packet (e.g., 143) just in time for itstransmission.

FIG. 6 shows a system to transmit data packets using multiple channelsaccording to one embodiment. For example, the system of FIG. 6 can beimplemented to use techniques of FIGS. 1, 2 and/or 3.

In FIG. 6, a scheduler (225) computes the sequences (235, 237, . . . ,239) of compositions of channel-encoded packets (e.g., 255, 257, . . . ,259) based on the number (221) of the original packets (121, 123, 125, .. . , 127) from which the channel-encoded packets (e.g., 255, 257, . . ., 259) are generated and the number (223) of the channels (105, 107, . .. , 109) that will be used for the transmissions of the channel-encodedpackets (e.g., 255, 257, . . . , 259).

For example, the compositions of initial channel-encoded packets of thesequences (235, 237, . . . , 239) can be determined using a method ofFIG. 4; and compositions of subsequent channel-encoded packets of thesequences (235, 237, . . . , 239) can be determined using a method ofFIG. 5.

Alternatively, the compositions of at least some of the initialchannel-encoded packets of the sequences (235, 237, . . . , 239) and/orthe subsequent channel-encoded packets of the sequences (235, 237, . . ., 239) can be determined based on a random process without using themethods of FIG. 4 and/or FIG. 5.

In FIG. 6, the sequences (235, 237, . . . , 239) of compositions areprovided to the transmitters (245, 247, . . . , 249) that are coupledwith the respective channels (105, 107, . . . , 109) for thetransmission of the respective channel-encoded packets (255, 257, . . ., 259). The transmitters (245, 247, . . . , 249) receive the same copyof the original packets (121, 123, 125, . . . , 127). Using therespective sequences (235, 237, . . . , 239) of compositions, thetransmitters (245, 247, . . . , 249) apply a predetermined function(e.g., XOR) to original packets identified in the respectivecompositions to generate the sequences of channel-encoded packets (255,257, . . . , 259) for transmission to the recipient device (229).

Preferably, each of the transmitters (245, 247, . . . , 249) generates asubsequent channel-encoded packet for a sequence (e.g., 235, 237, . . ., or 239) after the channel-encoded packets scheduled before thesubsequent channel-encoded packet in the sequence have been successfullytransmitted through the respective channel (e.g., 105, 107, . . . ,109).

Alternatively, channel-encoded packets (e.g., 255, 257, . . . , 259) maybe pre-generated before the transmission. For example, the scheduler(225) may generate the sequences of the channel-encoded packets (e.g.,255, 257, . . . , 259) and provide the channel-encoded packets (e.g.,255, 257, . . . , 259) directly to the transmitters (e.g., 245, 247, . .. , 249), instead of providing the combination of the original packets(121, 123, 125, . . . , 127) and the sequences (235, 237, . . . , 239)of compositions of the channel-encoded packets (255, 257, . . . , 259).In some instances, some of the transmitters (e.g., 247) are configuredto receive the sequence channel-encoded packets (e.g., 257), while othertransmitters (e.g., 245) are configured to receive the sequences (e.g.,235) of compositions and the original packets (e.g., 121, 123, 125, . .. , 127) to generate the channel-encoded packets (e.g., 255) on-the-flyduring their transmission.

In FIG. 6, the transmitters (245, 247, . . . , 249) use the channels(105, 107, . . . , 109) to transmit the channel-encoded packets (255,257, . . . , 259) in parallel. However, due to the real-time performancevariations, the sequence of the received packets (227) obtained by therecipient device (229) from the channels (105, 107, . . . , 109) may bepartially randomized. When the recipient device (229) receives asufficient number of channel-encoded packets via any combination of thechannels (105, 107, . . . , 109), the recipient device (229) can fullyrecover the sequence of original packets (121, 123, 125, . . . , 127)using the forward error correction technique for channel decoding.

The system of FIG. 6 allows the transmitters (245, 247, . . . , 249) totransmit their respective sequences of encoded packets (255, 257, . . ., 259) without coordination with each other. The sequences (e.g., 235,237, . . . , 239) of the encoded packets (e.g., 255, 257, . . . , 259)are predetermined and are not changed during the transmission. Thus,there is no need for reverse communications from the recipient device(229) to the transmitters (245, 247, . . . , 249) and/or the scheduler(225) for the coordination of the generation of the channel-encodedpackets across the channels (105, 107, . . . , 109).

Optionally, the recipient device (229) individually requests at leastsome of the transmitters (245, 247, . . . , 249) to terminate theirtransmissions, after the set of received packets (227) is sufficient torecover the original sequence of the original packets (121, 123, 125, .. . , 127) and before the respective transmitters (245, 247, . . . ,249) complete the transmission of their complete sequences (235, 237, .. . , 239) of encoded packets.

The recipient device may deliver any received original packets to anapplication (e.g., a video player) in order, determine whether any ofthe received channel-encoded packet is useful, and if enough usefulpackets have been received, it causes the termination of thetransmission across the parallel channels (105, 107, . . . , 109), andapplies FEC decoding to recover all of the original packets (121, 123,125, . . . , 127).

FIG. 7 shows a method to generate a channel-encoded packet according toone embodiment. For example, the method of FIG. 7 can be used by thescheduler (225) of FIG. 6 in the determination/selection (167) of thecomposition (166) of a packet (143) illustrated in FIG. 5, or thegeneration of the subsequent packet (135) illustrated in FIG. 3.

In FIG. 7, the scheduler (225) is configured to: determine (171) a firstnumber of packets (e.g., 131, 133) scheduled to be transmitted in afirst channel (e.g., 105) before a channel-encoded packet (135); anddetermine (173) a second number of packets (e.g., 141, . . . , 151)scheduled to be transmitted in second channels (e.g., 107, . . . , 109).The first number of packets (e.g., 131, 133) and the second number ofpackets (e.g., 141, . . . , 151) form a base set (161), where thepackets (e.g., 131, 133, 141, . . . , 151) in the base set (161) areless than the total number of original packets (e.g., 121, 123, 125, . .. , 127) to be transmitted via the first and second channels (e.g., 105,107, . . . , 109).

The scheduler (225) identifies (175) a randomly selected subset of theoriginal packets (121, 123, . . . , 127) for the generation of theencoded packet (e.g., 135) according to a forward error correctioncoding method. If it is determined (177) that the subset is linearlydependent of the base set (161) (e.g., based on their compositionvectors), the subset is rejected; and the computing device identifies(175) another randomly selected subset, until it is determined (177)that the subset is linearly independent of the packets (e.g., 131, 133,141, . . . , 151) in the base set (161). The identification of thesubset provides the composition of the encoded packet (e.g., 135)scheduled to be transmitted after the first number of packets (e.g.,131, 133) in the first channel (e.g., 105).

In FIG. 7, the scheduler (225) (or the transmitters (245, 247, . . . ,249), or another device) generates (179) the encoded packet (135) from alinear combination (e.g., XOR) of the subset for transmission in thefirst channel (105) after the first number of packets (e.g., 131, 133).

FIG. 8 shows a method to transmit a data segment according to oneembodiment. For example, the method of FIG. 8 can be implemented in asystem illustrated in FIG. 6.

In FIG. 8, a scheduler (225) is configured to schedule (181) thetransmission of a sequence of original packets (121, 123, . . . , 125,127) in the plurality of channels (105, 107, . . . , 109) bydistributing the original packets (121, 123, . . . , 125, 127) to thechannels using a round-robin method (163). Preferably, the originalpackets (121, 123, . . . , 125, 127) distributed to the plurality ofchannels (105, 107, . . . , 109) function as the systematicchannel-encoded packets scheduled to be transmitted in the respectivechannels before the transmission of other subsequent non-systematicchannel-encoded packets.

In FIG. 8, for each subsequent packet (e.g., 135) to be transmitted ineach respective channel (e.g., 105) of the channels, the scheduler (225)identifies a base set (e.g., 161) of packets (e.g., 131, 133, 141, . . ., 151), which may include all packets (e.g., 131, 133) that arescheduled for transmission in the respective channel (e.g., 105) beforethe subsequent packet (e.g., 135) and optionally one or more packets(e.g., 141, . . . , 151) scheduled for parallel transmission in each ofthe channels (e.g., 107, . . . , 109) other than the respective channel(e.g., 105) for the subsequent packet (e.g., 135).

In FIG. 8, the scheduler (225) identifies (185), for the subsequentpacket (e.g., 135), a subset of the original packets (e.g., 123, . . . ,125, 127 as identified by the composition (166) of a subsequent packet(e.g., 143) in FIG. 5), such that the composition vector of the subsetis linearly independent of the composition vectors of the packets in thebase set. The subset can be randomly selected according to a forwarderror correction technique until the subset has a composition that islinearly independent of the compositions of the packets in the base set.

In FIG. 8, the scheduler (225) stores (187), for the subsequent packet(e.g., 135), the identifications of the subset of the original packetsas the composition of the subsequent packet (e.g., 135).

After the original packets (121, 123, 125, . . . , 127) are received(189), the scheduler (225) (or a transmitter (e.g., 245), or anotherdevice) linearly combines (191) the subset of the original packetsaccording to the stored identifications, or the stored composition ofthe subsequent packet (e.g., 135), to generate the subsequent packet(e.g., 135).

A transmitter (e.g., 245) transmits (193), in the respective channel(e.g., 105), the subsequent packet (e.g., 135) after the transmission ofthe packets (e.g., 131, 133) that are scheduled for transmission in therespective channel (e.g., 105) before the subsequent packet (e.g., 135).

In one aspect, a method implemented in a computing apparatus includes:

-   -   generating, by the computing apparatus, a plurality of first        data packets (121, 123, 125, . . . , 127) of a same length from        a data segment (111) to be transmitted via a plurality of        parallel communication channels (105, 107, . . . , 109); and        identifying, by the computing apparatus (e.g., using the        scheduler (225)) from the first data packets (121, 123, 125, . .        . , 127), a plurality of packet sequences (115, 117, . . . ,        119) for the plurality of parallel communication channels (105,        107, . . . , 109) respectively. Each respective sequence (e.g.,        115) of the plurality of packet sequences includes: one or a        plurality of initial packets (e.g., 131) that are original        packets (121, 123, 125, . . . , 127) and that are scheduled to        be transmitted before subsequent packets (e.g., 133, 135, . . .        , 139) in the respective sequence (e.g., 115); and subsequent        packets (e.g., 135) that are linear combinations of a subset of        the original packets (e.g., 121, 123, . . . , 127). The method        further includes transmitting, by the computing apparatus (e.g.,        using the transmitters (245, 247, . . . , 249)), the plurality        of sequences (115, 117, . . . , 119) of packets in the plurality        of parallel communication channels (105, 107, . . . , 109)        respectively.

During the transmitting, a recipient device (229) coupled to theplurality of parallel communication channels (105, 107, . . . , 109) isconfigured to determine whether a received set of packets (227),received via the parallel communication channels (105, 107, . . . ,109), is sufficient to reconstruct the data segment (111). In responseto a determination that the received set of packets (227) is sufficientto reconstruct the data segment (111), the recipient device (229)directly terminates the transmitting of the sequences of channel-encodedpackets. Alternatively, the recipient device (229) transmits anindicator to the computing apparatus to stop the transmitting of thesequences of channel-encoded packets.

In one embodiment of the aspect, the subsequent packet (135) isgenerated according to a forward error correction technique, where arandomly selected subset of the original packets (121, 123, 125, . . . ,127) are combined via a predetermined function (e.g., a linearcombination, an exclusive or (XOR) operation).

In one embodiment of the aspect, the method includes distributing theplurality of original packets (e.g., 121, 123, 125, . . . , 127) to theplurality of packet sequences (e.g., 115, 117, . . . , 119) for theplurality of parallel communication channels (e.g., 105, 107, . . . ,109) respectively according to a round-robin schedule (163).Alternatively, the method includes distributing the plurality oforiginal packets (e.g., 121, 123, 125, . . . , 127) to the plurality ofpacket sequences (e.g., 115, 117, . . . , 119) for the plurality ofparallel communication channels (e.g., 105, 107, . . . , 109)sequentially.

In one embodiment of the aspect, the subsequent packet (e.g., 135) isselected to be linearly independent of packets (e.g., 131, 133) thatinclude the original packet (e.g., 131) and that are scheduled to betransmitted in a respective channel (e.g., 105) before the subsequentpacket (135). The subsequent packet may be further selected to belinearly independent of further packets (e.g., 141, . . . , 151) thatare transmitted in the plurality of parallel communication channels(e.g., 107, . . . , 109) other than the respective channel (e.g., 105)in which the subsequent packet (135) is transmitted.

In another aspect, a method implemented in a computing apparatusincludes: generating, by the computing apparatus, a plurality oforiginal packets (121, 123, 125, . . . , 127) of a same length from adata segment (111) to be transmitted via a plurality of parallelcommunication channels (105, 107, . . . , 109); and identifying, by thecomputing apparatus (e.g., using the scheduler (225)) from the originalpackets, a plurality of packet sequences (115, 117, . . . , 119) for theplurality of parallel communication channels (105, 107, . . . , 109)respectively. Each respective sequence (e.g., 115) of the plurality ofpacket sequences includes a subsequent packet (e.g., 135) that isscheduled to be transmitted in a respective channel (e.g., 105) of theplurality of parallel communication channels and that is linearlyindependent of packets (e.g., 131, 133) that are scheduled to betransmitted in the respective channel (e.g., 105) before the subsequentpacket (e.g., 135). Further, the subsequent packet (e.g., 135) is alinear combination of a subset of the original packets (e.g., 121, 123,125, . . . , 127). The method further includes transmitting, by thecomputing apparatus (e.g., using the transmitters (245, 247, . . . ,249)), the plurality of sequences (115, 117, . . . , 119) of packets inthe plurality of parallel communication channels (105, 107, . . . , 109)respectively. During the transmitting, a recipient device (229) coupledto the plurality of parallel communication channels (105, 107, . . . ,109) is configured to determine whether a received set of packets (227),received via the parallel communication channels (105, 107, . . . ,109), is sufficient to reconstruct the data segment (111); and inresponse to a determination that the received set of packets (227) issufficient to reconstruct the data segment (111), the recipient device(229) directly terminates the transmitting of the sequences ofchannel-encoded packets. Alternatively, the recipient device (229)transmits an indicator to the computing apparatus to stop thetransmission of the remaining parts of the sequences (115, 117, . . . ,119).

In one embodiment of this aspect, the subsequent packet (135) is furtherselected to be linearly independent of further packets (e.g., 141, . . ., 151) that are transmitted in the plurality of parallel communicationchannels (107, . . . , 109) other than the respective channel (105) inwhich the subsequent packet (135) is transmitted. For example, thefurther packets (e.g., 141, . . . , 151) are evenly distributed in theplurality of parallel communication channels (e.g., 107, . . . 109)other than the respective channel (105) in which the subsequent packet(135) is transmitted.

In one embodiment of this aspect, the further packets (141, . . . , 151)for the other channels (e.g., 107, . . . , 109) and the packets (e.g.,131, 133) that are scheduled to be transmitted in the respective channel(105) before the subsequent packet (135) form a base set (161) ofpackets; the base set (161) is selected to have fewer packets than theplurality of original packets (121, 123, 125, . . . , 127); and thesubsequent packet (135) is selected to be linearly independent of thebase set of data packets.

In one embodiment of this aspect, the subsequent packet (e.g., 135) isgenerated according to a forward error correction technique, whichrandomly selects the subset of the original packets (121, 123, 125, . .. , 127) according to a predetermined distribution, and combines thesubset of the original packets (121, 123, 125, . . . , 127) using alinear function (e.g., exclusive or (XOR)). The random selection isrepeated until the selected subset is linearly independent of the baseset (161).

In one embodiment of this aspect, the method further includes:distributing the plurality of original packets (121, 123, 125, . . . ,127) to the plurality of packet sequences (115, 117, . . . , 119),according to a round-robin schedule (163), as initial packets of thepacket sequences (115, 117, . . . , 119). Alternatively, the methodfurther includes: distributing the plurality of original packets (121,123, 125, . . . , 127) to the plurality of packet sequences (115, 117, .. . , 119) sequentially as initial packets of the packet sequences (115,117, . . . , 119).

In a further aspect, a method implemented in a computing apparatusincludes: identifying, by the computing apparatus (e.g., using thescheduler (225), compositions (e.g., 166) of a plurality of sequences(115, 117, . . . , 119) of packets to be transmitted respectively via aplurality of parallel communication channels (e.g., 105, 107, . . . ,109) to communicate a predetermined number of original packets (121,123, 125, . . . , 127) of a same length to a recipient device (229). Acomposition (e.g., 166) of each respective packet (e.g., 143) in theplurality of sequences of packets includes an identification of eachpacket, among the original packets (121, 123, 125, . . . , 127), that isto be used to generate the respective packet (e.g., 143); and eachrespective sequence (e.g., 117) of the plurality of sequences includesat least one subsequent packet (143) that is a linear combination of asubset of the original packets (e.g., 121, 123, . . . , 125, 127)identified by a composition (e.g., 166) of the subsequent packet (143)to enable forward error correction. The computing apparatus stores thesequences (235, 237, . . . , 239) of compositions of the channel-encodedpackets (e.g., 131, 133, 135, . . . , 139; 141, 143, 145, . . . , 149;151, 153, 155, . . . , 159). After receiving a data segment (111), thecomputing apparatus generates the predetermined number of originalpackets (121, 123, 125, . . . , 127) from the data segment (111) andretrieves the sequences (235, 237, . . . , 239) of compositions togenerate of the channel-encoded packets from the original packets (121,123, 125, . . . , 127) according to the compositions. The computingapparatus uses the transmitters (235, 237, . . . 239) to transmit theplurality of sequences of channel-encoded packets (255, 257, . . . ,259) in the plurality of parallel communication channels (105, 107, . .. , 109) respectively. During the transmitting, the recipient device(229) determines whether a current set of received packets (227),received via the parallel communication channels (105, 107, 109), issufficient to reconstruct the data segment (101); and if so, therecipient device (229) either directly terminates the transmission ofremaining packets in each of the channels (105, 107, . . . , 109), ortransmits an indicator to the computing apparatus (e.g., via thetransmitters 245, 247, . . . , 249) to stop the transmission ofremaining packets in each of the channels (105, 107, . . . , 109).

In one embodiment of this aspect, the identification of the sequences(115, 117, . . . , 119) includes initially distributing the plurality oforiginal packets (121, 123, 125, . . . , 127) to the plurality ofsequences (115, 117, . . . , 119) according to a round-robin orsequential schedule (163) as the initial packets of the sequences (115,117, . . . , 119).

In one embodiment of this aspect, the channel-encoded packets aregenerated from the original packets (121, 123, 125, . . . , 127)according to the compositions based on an exclusive or (XOR) operationapplied on subsets of the original packets (121, 123, 125, . . . , 127)identified by the compositions of the respective channel-encodedpackets.

In one embodiment of this aspect, the composition (166) of thesubsequent packet (143) is a randomly selected subset of the originalpackets (121, 123, 125, . . . , 127); and the randomly selected subsetis selected such that it has a composition that is linearly independentof the compositions of: packets (123) that are queued in the respectivesequence (117), in which the subsequent data packet (143) belongs,before the subsequent data packet (143); and one or more initial packets(121, 125, . . . ) that are queued in plurality of sequences (115, 119,. . . ) other than the respective sequence (117) in which the subsequentdata packet (143) belongs.

The scheduler (225), the transmitters (245, 247, . . . , 249), therecipient device (229) and/or other devices used in the system of FIG. 6and/or the methods discussed above can be implemented using a dataprocessing system as illustrated in FIG. 13, with more or lesscomponents.

The performance of the communication systems discussed above (e.g., asillustrated in FIG. 1, 2, 3 or 6), or other communication systems thatuse forward error correction, can be optimized or improved for thetransmission of a given data segment (111) by adjusting/optimizing thenumber of packets (121, 123, 125, . . . , 127) into which the datasegment (111) is partitioned.

When the data segment (111) is partitioned into different numbers ofpackets (121, 123, 125, . . . , 127) for transmission using a forwarderror correction technique, the transmission has different datacommunication overheads and/or computation overheads. The cost of theoverheads can be evaluated as a function of the number of packets usedin partitioning the data segment (111) of a given size. An optimizednumber of packets used in partitioning the data segment (111) is thendetermined from the cost function.

FIG. 9 illustrates an overhead cost in data communication fortransmitting a data segment by dividing the data segment into aplurality of data packets for forward error control.

In FIG. 9, the data segment (111) has a data size of L (e.g., measuredin bytes when the smallest unit of data communication in the system is abyte). When the data segment (111) is partitioned into K packets of anequal size P, it is typical that the sum of sizes (e.g., K*P) of thepackets (121, 123, 125, . . . , 127) does not equal to the size L of thedata segment (111); and as a result, some or all of the packets (121,123, 125, . . . , 127) have dummy data (e.g., with zeros, ones, orrandom data) added therein to ensure that the packets (121, 123, 125, .. . , 127) have the same size P. The sum of the overheads (231, 233,235, . . . , 237) in the data packets (121, 123, 125, . . . , 127) dueto the padding provides the size of the data overhead (241) (e.g.,K*P−L).

When the data (221, 223, 225, . . . , 227) of the data segment (111) isdistributed to K data packets (121, 123, 125, . . . , 127), the minimumsize P for the data packets (121, 123, 125, . . . , 127) is the smallestinteger that is greater than or equal to L/K.

The data (221, 223, 225, . . . , 227) of an equally length can bedistributed from the data segment (111) to the packets (121, 123, 125, .. . , 127) such that the overheads (231, 233, 235, . . . , 237) in thepackets (e.g., in the form of padded zeros, ones or random data) have anequal length.

Alternatively, different packets (e.g., 121, 123, 125, . . . , 127) mayhave overheads (231, 233, 235, . . . , 237) of different sizes. Forexample, each of the packets (121, 123, 125, . . . ) except the lastpacket (127) has no overhead in one arrangement; and the last packet(127) carries the entire data overhead (241) (e.g., K*P−L, where P is atleast the smallest integer that is greater than or equal to L/K).

Transmitting the data overhead (241) is the extra cost for transmittingthe data segment (111) by dividing the data segment (111) into thepackets (121, 123, 125, . . . , 127) of the same length for channelencoding using a forward error correction technique.

The original packets (121, 123, 125, . . . , 127) having the dataoverhead (241) can be used to generate the packets in the copies (115,117, . . . , 119) for transmission over parallel channels (105, 107, . .. , 109), as illustrated in FIGS. 1-8.

FIG. 10 illustrates the data structure of a communication packet of oneembodiment.

For example, the communication packet (243) can be used to transmit atypical packet in the copies (115, 117, . . . , 119) illustrated in FIG.1, 2, or 3, or one of the channel-encoded packets (225, 257, . . . ,259) illustrated in FIG. 6.

In FIG. 10, the communication packet (243) includes a plurality of datafields to provide information that can be used for decoding. Forexample, the communication packets (243) includes fields specifying thenumber of packets K (251), the number of padded zeros (or ones, orrandom data, or dummy data) (253) in the coded data payload (257), andthe coefficient vector (255) (e.g., the composition (166) of achannel-encoded packet (143)) used to generate the coded data payload(257) from the packets (121, 123, 125, . . . , 127), in addition to thefield for the coded data payload (257) that has the size P.

The coefficient vector (255) identifies the composition vector (e.g.,166) of a channel-encoded packet (243). The first K bits of thecoefficient vector (255), as identified by the field of number ofpackets K (251), correspond to the K original packets (121, 123, 125, .. . , 127) respectively. When the i′th bit in the coefficient vector(255) has a first value (e.g., 1), the i′th packet in the packets (121,123, 125, . . . , 127) is used to generate the coded data payload (257)transmitted via the communication packet (243); and when the i′th bit inthe coefficient vector (255) has a second value (e.g., 0), the i′thpacket in the packets (121, 123, 125, . . . , 127) is not used togenerate the coded data payload (257) transmitted via the communicationpacket (243). The data fields (251, 253, and 255) for providing theinformation to assist the recipient device (229) in decoding the codeddata payloads (e.g., 257) are the overhead (245) in the communicationpacket (243).

The number of packets K (251) and the number of padded zeros (ones,random data, or dummy data) (253) in the coded data payload (257) can betransmitted for each communication packet used to transmit a packet inthe copies (e.g., 115, 117, . . . , 119). In other implementations, thenumber of packets K (251) and the number of padded zeros (or ones orrandom data) (253) in the coded data payload (257) are transmitted onlyonce for each of the copies (e.g., 115, 117, . . . , 119) to reduceoverhead. In other implementations, additional data fields may beincluded for each communication packet (243).

The number of packets K (251) is typically limited to be less than apredetermined limit (e.g., 256). Thus, the size of data field for thenumber of packets K (251) is predetermined (e.g., 1 byte) does notchange as a function of the number of packets K (251). Further, the datafield for the number of padded zeros (or ones or random data) (253) hasa predetermined size (e.g., 4 bytes) that is not a function of thenumber of packets K (251).

However, the size of the field of the coefficient vector (255) typicallychanges as a function of the number of packets K (251). For example, theminimum size Q of the coefficient vector (255) (e.g., in bytes) is equalto the smallest integer that is greater than or equal to K/8.

FIG. 11 illustrates various cost considerations in the transmission of adata segment using a forward error correction technique.

In FIG. 11, partitioning (101) the data segment (111) having a size Linto the original packets (121, 123, 125, . . . , 127) having an equalsize P results in the data overhead (241) as illustrated in FIG. 9. Thecost of transmitting the padded zeros (ones, random data, or dummy data)corresponding to the transmission cost of the data overhead (241).

In FIG. 11, transmitting each packet (e.g., 131, 133, 135, . . . , 139;141, 143, 145, . . . , 149; . . . ; 151, 153, 157, . . . , or 159) usingthe forward error correction technique results in the transmission ofadditional data fields (e.g., 251, 253, 255 illustrated in FIG. 10) andthus the overhead (245). The cost of transmitting the overhead in eachcommunication packet (243) used to carry a respective data packet in thecopies (115, 117, . . . , 119) corresponds to the cost of transmittingcommunication packet overhead.

In FIG. 11, there is a computation cost of the forward error correctiontechnique in encoding (103) the original packets into the packets (115,117, . . . , 119) and/or decoding the copies to generate the recovereddata segment (113). The computation cost is typically a function of thenumber of original packets K (251) into which the data segment (111) ispartitioned. When fountain codes are used in FEC, the cost of thecomputational complexity is typically in the order of log (K).Alternatively, the cost function of computational complexity may beempirically measured, or derived from curving a set of data points.

Further, due to the uncertainty in the communication system (e.g.,random delay in the channels), the actual number of packets therecipient device (229) will receive from each of the channels (105, 107,. . . , 109) is random before the recipient device (229) obtainssufficient packets for decoding (104) and then prevents the furthertransmission of the remaining packets. This introduces uncertainty inusefulness of the received packets, which causes that more than Kcommunication packets are communicated via the channels (105, 107, . . ., 109) as a whole before the recipient device (229) receives asufficient number (e.g., K) of useful packets for the generation of therecovered data segment (113) and thus terminates the transmission. Theexact number of additional communication packets transmitted via thesystem changes as the uncertainty in the communication system (e.g.,random delays in the channels in transmitting the packets of thecopies). An estimated (or average number, or an upper limit) of theadditional communication packets over the number of packets K can beused to determine the cost of transmitting redundant packets (not usefulpacket for decoding the channel-encoded packets).

For example, when the copies (115, 117, 119) are generated using around-robin (or sequential) allocation (163) scheme (e.g., asillustrated in FIG. 4) with subsequent channel-encoded packets generatedto be linearly independent of their base sets (e.g., as illustrated inFIG. 5), no more than K+d communication packets (e.g., d=3) are expectedto be transmitted in the communication system over the parallelcommunication channels (1105, 107, . . . , 109), before the transmissioncan be terminated. Thus, the cost of transmitting the redundant packetscan be estimated as the size of the additional d communication packets.

For example, when the data segment of a size L is partitioned (101) intoK original packets (121, 123, 125, . . . , 127), the minimal size P ofeach original packets (121, 123, 125, . . . , 127) is the smallestinteger that is large than or equal to L/K. Thus, the field for codeddata payload (257) of the communication packet has a size of P. Thefield for the coefficient vector (255) has a size of Q which is at leastthe smallest integer that is larger than or equal to K/8. When thecommunication packet uses one byte to transmit the number of packets(251), four bytes to transmit the number of padded zeros (253) (orpadded ones, random data or dummy data), the size of the communicationpacket (243) is (5+Q+P). When K+d (e.g., d=3) of communication packets(243) are used to transmit the data segment in order for the recipientdevice (229) to generate the recovered data segment (113), the totalsize of the transmitted communication packets is (5+Q+P)*(K+d); and thebandwidth cost, including the cost for transmitting the data overhead(241), the cost for transmitting the communication packet overhead(245), and cost for transmitting the redundant packets transmittedbefore the termination, is proportional to (5+Q+P)*(K+d). The cost ofFEC encoding (103) and/or decoding (104) is typically proportional toL*log (K).

The cost of FEC encoding (103) and/or decoding (104) and the cost of thecommunication (e.g., data overhead (241), communication packet overhead(245), redundant packets) may have different weight. Different weightscan be assigned to the computation cost and the communication cost togenerate a combined cost function of the number K to partition (101) thedata segment (111) into the original K packets (121, 123, 125, . . . ,127). An optimized K can be determined from the cost function for thepartition (101) of the data segment (111).

FIG. 12 shows a method to transmit a data segment according to oneembodiment.

In FIG. 12, after receiving (261) the size L of a data segment (111) tobe divided into original packets (121, 123, 125, . . . , 127) fortransmission using a forward error correction (FEC) coding technique, acomputing device is configured to compute (263) a cost of transmittingthe data segment (111) using the FEC coding technique by dividing thedata segment (111) into a predetermined number K of original packets(121, 123, 125, . . . , 127). The cost may include the communicationcost of the data overhead (241) as illustrated in FIG. 9, thecommunication cost of the overhead (245) in communication packet (243)as illustrated in FIG. 10, the computing cost of the FEC codingtechnology as illustrated in and discussed in connection with FIG. 11,and/or the communication cost of the redundant packets, as illustratedin and discussed in connection with FIG. 11.

In FIG. 12, the computing device adjusts (265) the number K to identifya number K′ to minimize the cost of transmitting the data segment.

After the number K′ is identified, the computing device divides (267)the data segment (111) of size L into the number K′ of original packets(121, 123, 125, . . . , 127), generates (269) channel-encoded packets(e.g., 131, 133, 135, . . . , 139; 141, 143, 145, . . . 149; . . . ;151, 153, . . . , 159) from the number K′ of original packets (121, 123,125, . . . , 127), and transmit (271) the channel-encoded packets (e.g.,131, 133, 135, . . . , 139; 141, 143, 145, . . . 149; . . . ; 151, 153,. . . , 159) using a plurality of parallel communication channels (105,107, . . . , 109).

For example, the method of FIG. 12 can be implemented in a systemillustrated in FIG. 11. The copies (115, 117, . . . , 119) of packets tobe transmitted via the parallel communication channels (105, 107, . . ., 109) can be generated using the techniques of FIG. 2 or 3, where thedistribution of systematic packets can be performed using the techniqueof FIG. 4, and the compositions of subsequent channel-encoded packetscan be identified using the technique of FIG. 5. Further, thecompositions can be pre-computed in a way as illustrated in FIG. 6.

In some implementations, an explicit expression is derived for the costas a function of the number K of original packets into which the datasegment (111) of the size L is partitioned (101). An explicit functioncan be derived to compute the optimized K′ as a function of the size Lof the data segment (111) to be transmitted.

In some implementations, a table of optimized K′ is pre-computed as afunction of the size L of the data segment (111) for a particularcommunication system (e.g., having its unique weight of thecommunication cost for communication over the channels (105, 107, . . ., 109) and the computation cost for FEC encoding (103) and/or decoding(104). Subsequently, the optimized K′ for a given size L of a particulardata segment (111) can be looked up from the table.

In some implementations, an empirical formula (e.g., generated fromcurve fitting the table of K′ vs. L) is used to compute the optimizednumber K′ of division from a given size L of the data segment (111).

In some implementations, the optimized K′ is computed just in time forthe transmission of the data segment (111) having the size L, in view ofthe cost characteristics of the communication system available totransmit the data segment (111) using a FEC technique.

In one aspect, a method implemented in a computing apparatus includes:receiving, in the computing apparatus, a size L of a data segment (111)to be transmitted by the computing apparatus; and determining, by thecomputing apparatus, a number K by optimizing a cost of transmitting thedata segment (111), such that the data segment (111) is divided into thenumber K of original packets (121, 123, 125, . . . , 127) based on whichchannel-encoded packets (e.g., 143 in FIG. 2 or 5), are generated fortransmission using a forward error correction technique. Anyoptimization techniques can be used to search for and/or compute theoptimal K from the function of the cost of transmitting the data segment(111).

In one embodiment of the aspect, after the optimal K is determined, thecomputing apparatus partitions (101) the data segment (101) having thesize L into the number K of original packets (121, 123, 125, . . . ,127), generates a plurality of copies (115, 117, . . . , 119) of packetsfrom the original packets (121, 123, 125, . . . , 127) using the forwarderror correction technique, and transmits, to a recipient device (229),the plurality of copies (115, 117, . . . , 119) of packets using aplurality of parallel communication channels (105, 107, . . . , 109)respectively. The recipient device (229) determines whether a receivedset of packets received via the parallel communication channels (105,107, . . . , 109) is sufficient to generate the recovered data segment(113), and if so, causes termination of the transmission of the copies(115, 117, . . . , 119), when sufficient numbers of useful packets havebeen received in the recipient device from the channels (105, 107, . . ., 109).

The cost of transmission considered for the optimization may include:first data transmission cost for dummy data (e.g., overhead 241) addedto the original packets (121, 123, 125, . . . , 127) so that theoriginal packets have a same size P; second data transmission cost fordata in communication fields (e.g., 251, 253, 255) added incommunication packets (e.g., 243) to enable decoding using the forwarderror correction technique (e.g., overhead 245); and/or third datatransmission cost for redundant communication packets that are expect tobe transmitted to the recipient device (229) before the termination ofthe transmission. The third data transmission cost can be estimatedbased on a count of maximum (or average, or expected) redundantcommunication packets that are expected to be transmitted via theparallel channels (105, 107, . . . , 109) before the termination of thetransmission of the copies (115, 117, . . . , 119). When the techniquesfor generating the copies (115, 117, . . . , 119) as illustrated inFIGS. 1-5 are used, the count is typically three using communicationchannels that each can transmit data reliably but may have differentand/or varying transmission speeds.

Further, the cost of transmission considered for the optimization mayinclude computation cost associated with application of the forwarderror correction technique, which is typically proportional to alogarithm function of the number K of original packets (121, 123, 125, .. . , 127).

Typically, the number K of original packets is limited to be no morethan 256. The cost for different values of K can be computedindividually to find the optimal K that minimizes the cost oftransmission.

When the technique of FIG. 4 is used, the plurality of copies (115, 117,. . . , 119) can be generated from the original packets (121, 123, . . ., 125, 127) by distributing the original packets (121, 123, . . . , 125,127) to the copies according to a round-robin scheme or a sequentialscheme as the initial/leading packets of the copies (115, 117, . . . ,119).

When the technique of FIG. 5 is used, the plurality of copies (115, 117,. . . , 119) can be generated from the original packets (121, 123, . . ., 125, 127) by selecting a subset of the original packets (e.g., asidentified by the composition (166) in FIG. 5) such that the subset islinearly independent on a base set (165) of packets in the copies (115,117, . . . , 119). A subsequent channel-encoded packet (e.g., 143) isgenerated from the subset for transmitting in a particular copy (117) ofthe plurality of copies (e.g., 115, 117, . . . , 119). The base setincludes one or more packets (e.g., 123) of the particular copy that arescheduled to be transmitted before the subsequent channel-encoded packet(143). The base set may further include at least initial packet (e.g.,packet (121) in copy (115), packet (125) in copy (119)) in each of theplurality of copies. Preferably, the subset does not include any oforiginal packets (e.g., 123) allocated to the particular copy (117).

For example, the subsequent channel-encoded packet (143) can begenerated from exclusive or of the packets in the subset; and the subsetis selected randomly according to a predetermined distribution. Therandom selection can be repeated until the selected subset is linearlyindependent of the base set (165) and/or does not include any of theoriginal packets allocated to the particular copy (117) and/or does notinclude any of the original packets in the base set (165).

The present disclosure includes the methods discussed above, computingapparatuses configured to perform methods, and computer storage mediastoring instructions which when executed on the computing apparatusescauses the computing apparatuses to perform the methods.

FIG. 13 shows a data processing system on which the methods of thepresent disclosure can be implemented. While FIG. 13 illustrates variouscomponents of a computer system, it is not intended to represent anyparticular architecture or manner of interconnecting the components. Oneembodiment may use other systems that have fewer or more components thanthose shown in FIG. 13.

In FIG. 13, the data processing system (200) includes an inter-connect(201) (e.g., bus and system core logic), which interconnects amicroprocessor(s) (203) and memory (211). The microprocessor (203) iscoupled to cache memory (209) in the example of FIG. 13.

In one embodiment, the inter-connect (201) interconnects themicroprocessor(s) (203) and the memory (211) together and alsointerconnects them to input/output (I/O) device(s) (205) via I/Ocontroller(s) (207). I/O devices (205) may include a display deviceand/or peripheral devices, such as mice, keyboards, modems, networkinterfaces, printers, scanners, video cameras and other devices known inthe art. In one embodiment, when the data processing system is a serversystem, some of the I/O devices (205), such as printers, scanners, mice,and/or keyboards, are optional.

In one embodiment, the inter-connect (201) includes one or more busesconnected to one another through various bridges, controllers and/oradapters. In one embodiment the I/O controllers (207) include a USB(Universal Serial Bus) adapter for controlling USB peripherals, and/oran IEEE-1394 bus adapter for controlling IEEE-1394 peripherals.

In one embodiment, the memory (211) includes one or more of: ROM (ReadOnly Memory), volatile RAM (Random Access Memory), and non-volatilememory, such as hard drive, flash memory, etc.

Volatile RAM is typically implemented as dynamic RAM (DRAM) whichrequires power continually in order to refresh or maintain the data inthe memory. Non-volatile memory is typically a magnetic hard drive, amagnetic optical drive, an optical drive (e.g., a DVD RAM), or othertype of memory system which maintains data even after power is removedfrom the system. The non-volatile memory may also be a random accessmemory.

The non-volatile memory can be a local device coupled directly to therest of the components in the data processing system. A non-volatilememory that is remote from the system, such as a network storage devicecoupled to the data processing system through a network interface suchas a modem or Ethernet interface, can also be used.

In this description, some functions and operations are described asbeing performed by or caused by software code to simplify description.However, such expressions are also used to specify that the functionsresult from execution of the code/instructions by a processor, such as amicroprocessor.

Alternatively, or in combination, the functions and operations asdescribed here can be implemented using special purpose circuitry, withor without software instructions, such as using Application-SpecificIntegrated Circuit (ASIC) or Field-Programmable Gate Array (FPGA).Embodiments can be implemented using hardwired circuitry withoutsoftware instructions, or in combination with software instructions.Thus, the techniques are limited neither to any specific combination ofhardware circuitry and software, nor to any particular source for theinstructions executed by the data processing system.

While one embodiment can be implemented in fully functioning computersand computer systems, various embodiments are capable of beingdistributed as a computing product in a variety of forms and are capableof being applied regardless of the particular type of machine orcomputer-readable media used to actually effect the distribution.

At least some aspects disclosed can be embodied, at least in part, insoftware. That is, the techniques may be carried out in a computersystem or other data processing system in response to its processor,such as a microprocessor, executing sequences of instructions containedin a memory, such as ROM, volatile RAM, non-volatile memory, cache or aremote storage device.

Routines executed to implement the embodiments may be implemented aspart of an operating system or a specific application, component,program, object, module or sequence of instructions referred to as“computer programs.” The computer programs typically include one or moreinstructions set at various times in various memory and storage devicesin a computer, and that, when read and executed by one or moreprocessors in a computer, cause the computer to perform operationsnecessary to execute elements involving the various aspects.

A machine readable medium can be used to store software and data whichwhen executed by a data processing system causes the system to performvarious methods. The executable software and data may be stored invarious places including for example ROM, volatile RAM, non-volatilememory and/or cache. Portions of this software and/or data may be storedin any one of these storage devices. Further, the data and instructionscan be obtained from centralized servers or peer to peer networks.Different portions of the data and instructions can be obtained fromdifferent centralized servers and/or peer to peer networks at differenttimes and in different communication sessions or in a same communicationsession. The data and instructions can be obtained in entirety prior tothe execution of the applications. Alternatively, portions of the dataand instructions can be obtained dynamically, just in time, when neededfor execution. Thus, it is not required that the data and instructionsbe on a machine readable medium in entirety at a particular instance oftime.

Examples of computer-readable media include but are not limited torecordable and non-recordable type media such as volatile andnon-volatile memory devices, read only memory (ROM), random accessmemory (RAM), flash memory devices, floppy and other removable disks,magnetic disk storage media, optical storage media (e.g., Compact DiskRead-Only Memory (CD ROM), Digital Versatile Disks (DVDs), etc.), amongothers. The computer-readable media may store the instructions.

The instructions may also be embodied in digital and analogcommunication links for electrical, optical, acoustical or other formsof propagated signals, such as carrier waves, infrared signals, digitalsignals, etc. However, propagated signals, such as carrier waves,infrared signals, digital signals, etc. are not tangible machinereadable medium and are not configured to store instructions.

In general, a machine readable medium includes any mechanism thatprovides (i.e., stores and/or transmits) information in a formaccessible by a machine (e.g., a computer, network device, personaldigital assistant, manufacturing tool, any device with a set of one ormore processors, etc.).

In various embodiments, hardwired circuitry may be used in combinationwith software instructions to implement the techniques. Thus, thetechniques are neither limited to any specific combination of hardwarecircuitry and software nor to any particular source for the instructionsexecuted by the data processing system.

Control of Multipath Data Transportation

FIG. 14 shows a multipath data transportation system having a controlunit implemented at a server side according to one embodiment. Thecontrol unit includes two primary components:

-   -   A load balancer that balances the load of each path for optimal        performance (e.g., high throughput, high robustness, low        redundancy);    -   A path predictor that predicts the quality (e.g., latency,        bandwidth, packet loss) of each path.

The load balancer solves an optimization problem of the multipath datatransportation system. Using this solution, the load balancer controlsthe encoding process that is applied to the data segment received by theserver. The encoding process generates communication packets in the formof copies, and which are distributed to paths 1, 2, . . . , 3 asdescribed below.

In one embodiment, the system generates multiple coded copies of a datasegment. Each copy will be downloaded through a different path. Thecontrol unit decides the size of each copy based on the quality of eachpath. The system is configured to 1) maximize aggregated throughput, 2)minimize redundancy, 3) maximize robustness, and 4) minimize decodingcomplexity.

As discussed further below, when determining the number of paths to usefor transmission, the number of used paths is first determined byassuming no transmission time window. It may then be further increasedif a transmission time window is imposed.

In one embodiment, feedback from the receiver is provided to the pathpredictor as shown in FIG. 14. In one embodiment, each path is one ofthe channels 105, 107, 109 of FIG. 1. In one embodiment, the dataprocessing system of FIG. 13 above is used to implement the multipathdata transportation system (including the control unit). In oneembodiment, the control unit is located in the server of FIG. 14. Inanother embodiment, the control unit is located in the receiver (e.g.,the recipient device of FIG. 1 above).

In one embodiment, a data segment is a video frame in a video streamingsystem. Such systems typically generate 30 to 60 video frames persecond. Correspondingly, a typical server is a video streaming server,and a typical receiver is a video player. One example is a cloud gamingsystem. In this system, the cloud gaming server executes the game usinga cloud GPU, renders the gaming video and sends video frames to theplayer computing device (receiver). The player receives these frames andplays the video. This way, the player does not need high-end hardware toexecute the game.

In one embodiment, as discussed in more detail below, an optimal loadK(n) is determined for each path when sending a data segment. K(n) isthe number of packets to be sent through path-n. By assuming N′ pathswith the shortest single-trip-time STTn are used, the goal is tocomplete the download of K(n) (n=1, 2, . . . , N′) at the same time.This download time for path-n is Tn=STTn+K(n)/BWPn, where BWPn is thebandwidth of path-n in terms of packets per second. By solving equationT1=T2= . . . =TN′ under the constraint that the sum of K(n) packetsallow the decoding of the data segment, K(n) can be determined. The N′and the corresponding set of K(n) that minimizes the download time isthe optimal solution, which may be further adjusted when constraintssuch as transmission time window apply.

Various terminologies, abbreviations, and notations are set forth belowas used in the following discussion of various embodiments of themultipath data transportation system.

Communication A packet generated from original data packets using apacket coding technique. There are two types of communication packets:systematic and coded. Coded packet A type of communication packet, it isthe binary XOR of a subset of the original data packets, plus overhead.Systematic An original data packet plus overhead. packet Copy The set ofcommunication packets sent through a path. It consists of somesystematic packets, followed by some coded packets.

BW Bandwidth FPS Frame-per-second HTTP Hypertext Transfer Protocol PLRPacket loss rate RTT Round-trip-time STT Single-trip-time TCPTransmission Control Protocol UDP User Datagram Protocol

BWn The bandwidth of path-n BWPn The number of communication packets theserver can send through path-n in one second BWPRn The number ofcommunication packets the receiver can receive from path-n in one secondC(n) The number of coded packets sent through path-n K The number oforiginal data packets in a data segment. It is also the minimum numberof communication packets the receiver must receive in order to decodethe data segment. K(n) The number of communication packets sent througheach path. K(n) = S(n) + C(n) Kcomp(n) The number of leftover packetscompensated by path-n Kleft The packets left over from path-n due to thetransmission time window Kmax(n) The maximum number of communicationpackets that can be sent through path-n due to the transmission timewindow N The number of paths PLRn The inherent and inevitable PLR ofpath-n due to noise, interference, etc. Pr The receiver observed PLR,including those due to noise and congestion. r The data transmissionrate (i.e., the rate at which data is pushed to the path). S(n) Thenumber of systematic packets sent through each path. Tw The transmissiontime window of the data segment, after which the segment will expire.STTn The STT of path-n T The data segment delivery time. It is the timebetween the server starts sending and the receiver receives enoughcommunication packets to decode. t(n) The time between the server startssending and the receiver finishes its reception of communication packetssent through path-n

In one embodiment, the multipath data transportation system sends a datasegment using multiple network paths. At a high level, it generatesdifferent copies of the original data segment using a coding technique,and then sends different copies through different paths. In oneembodiment, channel encoding (103) of original packets (121, 123, 125, .. . , 127) that generates different channel-encoded copies (115. 117, .. . , 119) as was described in FIG. 1 above is used to provide thedifferent copies for sending through the different paths. The codingtechnique allows the receiver to recover the original data segment ifthe total amount of datagram received from all paths is equal to orslightly greater than the size of the original segment.

Specifically, a data segment (e.g., data segment 111 of FIG. 1) ispartitioned into K equal-length original data packets (e.g., packets121, 123, 125, and 127 of FIG. 1). These original data packets areencoded into a minimum of K communication packets (e.g., Packets 1.1,1.2, . . . , 1.k(1) as shown in FIG. 14) using the coding technique. Itshould be noted that in other embodiments different coding techniquesfrom those described above may be used. For example, other binary codessuch as low-density-parity-check (LDPC) code orBose-Chaudhuri-Hocquenghem (BCH) code with sufficiently large codingsizes could be used.

There are two types of communication packets:

-   -   Systematic packet: Each systematic packet is one of the K        original data packets, plus a header (also referred to herein as        overhead). Thus, the coding technique will generate a total of K        systematic packets, and this number is fixed regardless of the        number of paths.    -   Coded packet: It is the binary XOR of multiple original data        packets, plus a header. The coding technique is able to generate        any number of coded packets.

Each communication packet is sent through one of the multiple paths. Acopy is the set of communication packets sent through the same path.When the total number of communication packets received by the receiverfrom all paths is K+δ, where δ≥0 is a small integer, the receiver candecode the K original data packets and, thus, recover the data segment.

For such a system, there are four performance metrics that areoptimized:

-   -   Throughput: minimize the time spent on the download;    -   Reliability: maximize the probability that the receiver receives        enough communication packets for successful decoding. This        metric is particularly important when the underlying        transportation layer protocol is unreliable (e.g., UDP);    -   Redundancy: minimize the redundant packets the receiver will        receive after decoding.    -   Decoding complexity: maximize the number of systematic packets        the receiver will receive before decoding, as these packets do        not require decoding.

Equivalently, the number of coded packets the receiver will receivebefore decoding is minimized.

As explained below, these performance metrics are dominated by the wayin which the communication packets are sent. Specifically, there are twotransmission parameters to optimize:

-   -   the number K(n) of communication packets to be sent through        path-n, for all the N paths. This number consists of the number        S(n) of systematic packets and the number C(n) of coded packets        to be sent through each path;    -   the transmission rate at which the coded packets should be sent.

To optimize these two parameters, the control unit of the multipath datatransportation system is used. The control unit includes two components,namely a load balancer and a path predictor, as discussed further below.

Overview of Control Unit

In one embodiment, a flow chart of operation for the control unit isillustrated in FIG. 15. The control unit has two core components:

-   -   Path predictor: It predicts the quality of each path, such as        its BW (bandwidth), RTT (round-trip-time), and PLR (packet loss        rate);    -   Load balancer: based on the prediction, it decides the copy size        (i.e., the number of communication packets) and transmission        rate for each path.

The load balancer is discussed in more detail below, including adiscussion of how to optimize the size K(n) of each copy. In oneembodiment, the transmission rate is simply set it to the BW of eachpath to maximize throughput. The path predictor is discussed later belowafter the discussion of the load balancer.

Load Balancer

In one embodiment, communication packets (including K systematic packetsand a certain number of coded packets) of a data segment are generatedand each communication packet is sent through one of the multiplenetwork paths. In this embodiment, the transmission rate is set to theBW of each path. The number of communication packets to be sent througheach path is designed to optimize throughput, redundancy, reliability,and decoding complexity. This is demonstrated in more detail below.

In the following discussion for FIGS. 16-19, it is assumed for purposesof illustration that N=2 paths are used. The 1st (respective 2nd) pathhas a single-trip-time (STT) of STT1 (respective STT2). Thus, after theserver (Tx) starting to send coded packets at time t1, the receiver (Rx)will start to receive from path-1 (respective path-2) at time t2=t1+STT1(respective t3=t1+STT2). For each path-n, S(n) systematic packets arefirst sent, followed by C(n) coded packets. Here, {S(n)} across all theN paths satisfy that:

${\sum\limits_{n = 1}^{N}{S(n)}} = K$

In other words, each path will deliver a different subset of the Ksystematic packets.

An ideal example of delivery is shown in FIG. 16, which assumes nopacket loss in any of the N paths. {S(n)} of both paths are optimized,so that the receiver can finish the download of both S(1) and S(2) atthe same time t4. Since the {S(n)} systematic packets carry the Koriginal data packets, the receiver can directly recover the datasegment without any decoding.

On the other hand, if {S(n)} are not optimized, the receiver willreceive coded packets before being able to recover the data segment.This will introduce decoding cost.

In a more general, practical scenario, there will be random packet loss.In this case, the systematic packets may be lost, and will becompensated by the coded packets, which constitute the second part of acopy. Thus, in general the download of copies that consists of bothsystematic and coded packets is considered.

In the ideal example of delivery shown in FIG. 17 (with PLRO≥0), {K(n)}of both paths are optimized, so that the two downloads will be completedat the same time t4, and the downloaded packets together are just enoughto decode the data segment. This way, the download time (t4−t2) isminimized, which maximizes the throughput. Moreover, since the receiverdoes not receive any more communication packets of this data segmentafter t4, the redundancy is minimized to zero.

However, if the number of communication packets sent through each pathis sub-optimal, the download of one copy from one path may be completedsooner than the other copy from the other path. In the example in FIG.18, the receiver completes the download of copy-1 at t4, and thedownload of copy-2 at t5. Between t4 and t5, path-1 does not contributeto the transportation of the data segment, indicating a deficiency in BWusage and a prolonged decoding time (t5−t2). This yields a suboptimaldata throughput.

In another example shown in FIG. 19, the server sends too manycommunication packets, such that the receiver still receives somecommunication packets of the data segment after the decoding. All thesepackets are redundant, and are a waste of BW resources. Therefore, thenumber of communication packets K(n) for each path should be optimized,along with the number S(n) of systematic packets within each K(n).

Moreover, {K(n)} must be further adjusted if the data segment has alimited life span, because this limitation will impose a limitedtransmission time window, denoted by Tw. For example, in cloud gamingvideo streaming with 30 fps (frame per second), each video frame is onlyuseful for 33 ms. Thus, the transmission time window is Tw=33 ms, whichmeans that {K(n)} must be small enough to be sent within 33 ms. Thisrestriction further hardens the optimization of {K(n)}.

Therefore, the optimizations of {K(n)} and {S(n)} as used in multipathbalancing are highly nontrivial. The optimization algorithm is describedbelow, along with its implementations. However, the multipath loadbalancing of the present disclosure is first contrasted with traditionalmultipath load balancing.

Benefits Over Prior Approaches

In various embodiments, the multipath load balancing described herein isdifferent from its traditional counterparts due to a fundamentaldifference in system model:

Traditional load balancing techniques splits a data stream intoindependent sub-streams, and route different sub-streams throughdifferent paths. This approach incurs packet reordering and bufferingproblems at the receiver side when the receiver stitches the sub-streamstogether in order. The problem becomes severe when the paths' qualitychanges quickly and when there are slow paths.

In the multipath load balancing of the present disclosure, however, allthe paths collaboratively transport the same data segment. Thus, thereceiver only needs a buffer of size equal to one data segment. Due tothe same reason, packet reordering becomes irrelevant.

Load Balancing Algorithm

This section mathematically describes a multipath load balancingalgorithm according to various embodiments, which optimizes the numberK(n) of communication packets (including S(n) systematic packets, andC(n) coded packets) of each path based on the STT, PLR, BW of each path,and the transmission time window Tw. The algorithm includes three steps:

-   -   Optimize the load K(n) of each path based on the STT, PLR, BW of        each path, and Tw of the data segment;    -   Optimize the number of systematic packets S(n) of each path;    -   Optionally, further increase each K(n) (i.e., send more        communication packets), to gain extra robustness against network        randomness.

Calculation of Optimal Load {K(n)}

It is assumed that the data segment is partitioned to K data packets,and there are N paths. The time t1 when the server starts to send is setto zero. BWPn denotes the number of communication packets the sender cansend through path-n per second (throughput), and BWPRn denotes thenumber of communication packets the receiver can receive from path-n persecond (goodput):

${{BWP}_{n} = \frac{{BW}_{n}}{{Packet}{Size}}}{{{BWPR}_{n} = {{BWP}_{n}*\left( {1 - {PLR}_{n}} \right)}},}$

where BWn is the BW of path-n with a unit of bytes per second, packetsize is the size of each communication packet with a unit of byte, andPLRn is the PLR of path-n.

K(n) denotes the number of communication packets to be sent throughpath-n. The sent packets will be received by the receiver (with someloss) after a period of:

${t(n)} = {{STT}_{n} + \frac{K(n)}{{BWP}_{n}}}$

The overall delivery time T, i.e., the time to deliver {K(n)}communication packets, is the longest period across all the paths thatare used:

$\begin{matrix}{T = {\max\left\lbrack {{t(n)},{\forall{{n:{K(n)}} > 0}}} \right\rbrack}} \\{= {\max\left\lbrack {{{STT}_{n} + \frac{K(n)}{{BWP}_{n}}},{\forall{{n:{K(n)}} > 0}}} \right\rbrack}}\end{matrix}$

A goal is to minimize T by optimizing {K(n)}. The optimization issubject to two constraints:

-   -   Transmission time window:

K(n)

K _(max)(n)=BWP _(n) *T _(w)

-   -   Decoding success: the receiver must receive at least K        communication packets to decode the data segment. Since path-n        will contribute K(n)*(1−PLRn) packets, the following constraint        applies:

${\sum\limits_{n = 1}^{N}{{K(n)}*\left( {1 - {PLR}_{n}} \right)}} \geqslant K$

Mathematically, the optimization problem is defined as:

${{{Minimize}:T} = {\max\left\lbrack {{{STT}_{n} + \frac{K(n)}{{BWP}_{n}}},{\forall{{n:{K(n)}} > 0}}} \right\rbrack}}{{Subject}{to}:}\left\{ \begin{matrix}{{{K(n)} \leqslant {{BWP}_{n}*T_{w}}},{\forall n},{{if}T_{w}{{exists}.}}} \\{{\sum\limits_{n = 1}^{N}{{K(n)}*\left( {1 - {PLR}_{n}} \right)}} \geq K}\end{matrix} \right.$

In this optimization, the following first guideline is used regardingpriority of the paths: If path-n with STTn is used, then all the otherpaths with a STT of smaller than STTn must be used as well to maximizethe throughput. This guideline holds because the receiver will alwaysstart receiving from paths with smaller STT. Thus, by the time thereceiver starts receiving from path-n, it will already have receivedsome data from all the paths that have a smaller STT than path-n. Thereis no reason to exclude these paths.

In addition to the above guideline, the following second guideline isused: If the delivery time Tis smaller than the STT of path-n, thenpath-n is useless (i.e., set K(n)=0).

Without loss of generality, it is assumed that STT1<STT2< . . . <STTN,i.e., the 1^(st) path is the shortest. Then, the optimization isachieved using an iterative algorithm. In each of the N′-th iteration(1≤N′≤N), there are three steps:

-   -   (a) Motivated by the first guideline above, the load {K(n)} is        optimized by assuming that the first N′ paths (i.e., those with        the smallest STT) are used and that there is no Tw;

(b) If there is Tw that caps the K(n) of some of the used paths, {K(n)}is further adjusted by relocating the leftover packets to other usedpaths;

(c) Motivated by the second guideline above, the delivery time Tiscalculated given {K(n)} and it is compared with the STT of path-(N′+1).The iteration is stopped if T is smaller than the STT of path-(N′+1).Otherwise, N′ is increased by one and the next iteration is entereduntil N′=N.

Steps (a) and (b) above are now described in more detail below.

Step (a) Given N′, Optimize {K(n)} without Limited Transmission TimeWindow

When the first N′(N′≤N) paths are used, the minimum T that allowsdecoding must satisfy:

${\sum\limits_{n = 1}^{N^{\prime}}{\left( {T - {STT}_{n}} \right)*{BWPR}_{n}}} = K$

Thus, T can be calculated as:

$T = \frac{K - {\sum\limits_{n = 1}^{N^{\prime}}{{STT}_{n}*{BWPR}_{n}}}}{\sum\limits_{n = 1}^{N^{\prime}}{BWPR}_{n}}$

Once T is obtained, the number K(n) of communication packets of eachused path is calculated as:

K(n)BWP _(n)*(T−STT _(n)), ∀_(n)∈[1, . . . ,N′]

On the other hand, all the remaining N-N′ paths have their K(n)=0.

Step (b) Further Adjust Loads Based on Budget

If the sender (e.g., server) only has a limited time window Tw to sendthis data segment, the following upper bound on {K(n)} must be imposed:

K(n)

K _(max)(n)=BWP _(n) *T _(w)

If for path-n its K(n) is larger than Kmax(n), then by reducing K(n) toKmax(n), the number of communication packets the receiver will receivefrom this path is reduced by (K(n)−Kmax(n))*(1−PLR(n)). To securesuccessful decoding, these leftover packets are relocated to the otherused paths that have budget, i.e., those with K(n)<Kmax(n). To minimizethe delivery time T, the number of packets relocated to each of thesepaths will be proportional to their BWPRn. In other words, paths withhigher goodput will help compensate for more leftover packets.

For example, assume the first N′ paths are used. Further assume that thefirst N*(N*<N′) paths have their K(n)>Kmax(n), and the remaining N′-N*paths have their K(n)<Kmax(n) and thus still have budget. Then the totalnumber of leftover packets, denoted by Kleft, is:

$K_{left} = {\sum\limits_{n = 1}^{N^{*}}{\left( {{K(n)} - {K_{\max}(n)}} \right)*\left( {1 - {PLR}_{n}} \right)}}$

These leftover packets are relocated to the remaining N′-N* paths byusing their BWPRn as weights.

The total weight is:

Total Weight=Σ_(n=N*+1) ^(N′)BWPR_(n)

The leftover packets compensated by path-n (N*<n≤N′) is:

${K_{comp}(n)} = \frac{K_{left}*{BWPR}_{n}}{{Total}{Weight}}$

For the receiver to receive these packets, the K(n) of path-n (N*<n<=N′)must be increased as follows:

${K(n)} = {{{K(n)} + \frac{K_{comp}(n)}{1 - {PLR}_{n}}} = {{K(n)} + \frac{K_{left}*{BWP}_{n}}{{Total}{Weight}}}}$

Note that this increase may exceed the budget of path-n, yielding newleftover packets. Thus, the above relocation scheme is applied to the N′paths again. The iteration will continue until there are no leftoverpackets or all paths are full. A flow chart of the relocation algorithmis illustrated in FIG. 20.

After adjusting {K(n)}, the delivery time T is updated using:

$T = {\max\left\lbrack {{{STT}_{n} + \frac{K(n)}{{BWP}_{n}}},{\forall{{n:{K(n)}} > 0}}} \right\rbrack}$

If T<STT−(N′+1) or N′=N, the iteration is stopped. Otherwise, N′ isincreased by one and the iteration is continued.

A flow chart of the complete {K(n)} optimization algorithm is shown inFIG. 21.

Calculation of Optimal Number of Systematic Packets {S(n)}

In various embodiments, there are various {S(n)} optimization strategiesthat serve different purposes. Two of these optimization strategies aredescribed below.

In some embodiments, it is desired to maximize systematic packetdelivery speed. Systematic packets carry original portions of the datasegment. Thus, they can directly recover the data segment without anydecoding. On the other hand, coded packets cannot be used for recoveryuntil a sufficient number of packets are received.

When the transmission time window is too small for fulldelivery/recovery and when a subset of the data segment is still useful,it is desirable to maximize the number of systematic packets deliveredduring this time (i.e., systematic packets delivery speed) for abest-effort partial recovery of the data segment. This is particularuseful if each data segment is a video frame, because given that it ishard to fully deliver the frame, it is not desirable to discard thisframe. Instead, it is beneficial to reconstruct the frame as much aspossible using the systematic packets, and then compensate for themissing part through artefacts.

Accordingly, in a first {S(n)} optimization strategy, the systematicpackets' delivery speed can be maximized by distributing the Ksystematic packets to the paths proportionally to their quality, i.e.:

${S(n)} = \frac{K*{BWPR}_{n}}{\Sigma_{\forall{{n:{K(n)}} > 0}}{BWPR}_{n}}$

In one embodiment, this is useful for data streaming within the datasegment.

In other embodiments, it is desired to minimize decoding computationalcomplexity by maximizing the number of systematic packets. Decodingconsumes computational resources and the battery of the receiver and,thus should be minimized for lightweight receivers such as a mobilephone. Decoding also introduces decoding delay and, thus should beminimized for delay-sensitive applications such as cloud gaming.Systematic packets can be directly used for the recovery of the datasegment without decoding. Thus, decoding computational complexity can beminimized by maximizing the number of systematic packets the receiverwill receive.

Accordingly, in a second {S(n)} optimization strategy, the decodingcomputational complexity can be minimized by maximizing the number ofsystematic packets the receiver will receive. To achieve this, all thesystematic packets are allocated to the used paths that have the lowestPLR. This can be done using a simple iterative algorithm: In the n-thiteration, the used path is picked with the n-th lowest PLR and the S(n)of the picked used path is set to K(n). The iteration continues untilall the K systematic packets are allocated. FIG. 22 shows a flow chartof this algorithm.

Consequently, there could be three types of used paths: several low PLRpaths are used to only send systematic packets, one low PLR paths areused to send both systematic and coded packets (due to there are notenough systematic packets left to fill the K(n) of this path), and allthe remaining paths are used to only send coded packets.

FIG. 23 shows a complete multipath load balancing algorithm according toone embodiment.

Addition of Extra Redundancy to {K(n)}

In various embodiments, {K(n)} can be further increased to useredundancy to compensate for network randomness. If a transmission timewindow Tw is imposed, the redundancy-added {K(n)} may be passed to therelocation algorithm above to make sure K(n)<=Kmax(n).

In some embodiments, as just mentioned above, it is desired to useredundancy to compensate for network randomness. Redundant packets arealways used as long as the network presents randomness, such as packetloss and congestion. This is because such randomness will reduce thenumber of packets the receiver will receive on time from each path. Theshortage in received packets can only be compensated by sendingredundant packets. Such redundant packets are thus compulsory.

In addition, network randomness also implies that the estimation on thequality of the paths (e.g., packet loss, STT, bandwidth) may not beaccurate. Therefore, it is beneficial to use “extra redundancy” tocompensate for estimation offset. These extra redundant packets areoptional as long as the estimation is accurate.

In one embodiment, after K(n) is calculated based on the estimation ofthe path quality, extra redundancy may be added to compensate forestimation offset by further increasing K(n). After such increase, itmay not be possible to deliver these K(n) packets within thetransmission time window. Thus, K(n) should be further adjusted usingthe relocation algorithm above that deals with the transmission timewindow. In one embodiment, the K(n) of unused paths may also be set to asmall nonzero value, so that the receiver can probe the quality of thesepaths (e.g., RTT, BW, and PLR) through downloading a small datagram fromthem.

Various Implementations

In various embodiments, the multipath load balancing can be implementedat either the server side and the receiver side. When implemented at theserver side, the server will determine {K(n)} and push the packets tothe receiver (such as UDP based transportations). When implemented atthe receiver, the receiver will determine {K(n)} and pull the packetsfrom the sever (such as HTTP based transportations).

The multipath load balancing can be implemented on top of both reliableprotocols (such as TCP) and unreliable protocols (such as UDP). The onlydifference is that PLR is set to zero when a reliable protocol is used.

Path Prediction

In various embodiments, when performing multipath load balancing, thequality of each path is predicted (e.g., such as its RTT/STT, BW, PLR,etc.). Path prediction techniques can vary from system to system. Forexample, since TCP is reliable, PLR prediction is not required inTCP-based systems. On the other hand, UDP-based systems require morecomplicated path prediction techniques.

An example of one prediction technique for UDP-based systems is nowdescribed. STT can be readily measured through pinging. Hence, thisexample only focuses on BW and PLR prediction.

UDP protocol is unreliable. If the transmission rate r does not exceedthe path's bandwidth BW, the UDP packets will subject to an inherentnetwork packet loss of PLR. This loss is due to factors such as networknoise and interferences and, thus, is inevitable. If the transmissionrate r exceeds BW, then on top of the inherence packet loss, all theexcessive packets will be lost as well. Therefore, the packet loss Probserved by the receiver is:

$P_{r} = \left\{ \begin{matrix}{{PLR},} & {r \leq {BW}} \\{\frac{r - {{BW}\left( {1 - {PLR}} \right)}}{r},} & {r > {BW}}\end{matrix} \right.$

Graphically, the relation between transmission rate r and the receiverobserved PLR is illustrated in one example in FIG. 24.

Therefore, Pr is a function of r based on PLR and BW. If there are twopairs of r and Pr, say (r1, Pr1) and (r2, Pr2), where r1<BW and r2>BW,BW and PLR can be predicted using standard regression techniques. Suchpairs can be obtained by a simple transmission rate control algorithmwhich increases r gradually until a significant increase in Pr isreported by the receiver.

The predicted BW and PLR of all paths, together with the measured STT,can then be used by the load balancer for multipath load balancing.

The control unit of the multipath data transportation system asdescribed above can be used in various embodiments for maximization ofthroughput, maximization of robustness, minimization of redundancy,and/or minimization of decoding complexity. It achieves one or more ofthese benefits by predicting the quality of each path and then balancingthe number of communication packets to be sent through each path. Forexample, the control unit can be used for a coding-aided multipath datatransportation system, and the load balancing algorithm solves anoptimization problem.

Other Aspects

The description and drawings are illustrative and are not to beconstrued as limiting. The present disclosure is illustrative ofinventive features to enable a person skilled in the art to make and usethe techniques. Various features, as described herein, should be used incompliance with all current and future rules, laws and regulationsrelated to privacy, security, permission, consent, authorization, andothers. Numerous specific details are described to provide a thoroughunderstanding. However, in certain instances, well known or conventionaldetails are not described in order to avoid obscuring the description.References to one or an embodiment in the present disclosure are notnecessarily references to the same embodiment; and, such references meanat least one.

The use of headings herein is merely provided for ease of reference, andshall not be interpreted in any way to limit this disclosure or thefollowing claims.

Reference to “one embodiment” or “an embodiment” means that a particularfeature, structure, or characteristic described in connection with theembodiment is included in at least one embodiment of the disclosure. Theappearances of the phrase “in one embodiment” in various places in thespecification are not necessarily all referring to the same embodiment,and are not necessarily all referring to separate or alternativeembodiments mutually exclusive of other embodiments. Moreover, variousfeatures are described which may be exhibited by one embodiment and notby others. Similarly, various requirements are described which may berequirements for one embodiment but not other embodiments. Unlessexcluded by explicit description and/or apparent incompatibility, anycombination of various features described in this description is alsoincluded here. For example, the features described above in connectionwith “in one embodiment” or “in some embodiments” can be all optionallyincluded in one implementation, except where the dependency of certainfeatures on other features, as apparent from the description, may limitthe options of excluding selected features from the implementation, andincompatibility of certain features with other features, as apparentfrom the description, may limit the options of including selectedfeatures together in the implementation.

The disclosures of the above discussed patent documents are herebyincorporated herein by reference.

In the foregoing specification, the disclosure has been described withreference to specific exemplary embodiments thereof. It will be evidentthat various modifications may be made thereto without departing fromthe broader spirit and scope as set forth in the following claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative sense rather than a restrictive sense.

What is claimed is:
 1. A method implemented in a computing apparatus,the method comprising: predicting, by the computing apparatus, a qualityfor each of a plurality of network paths between the computing apparatusand one or more server devices storing a data segment comprising anumber of original data packets; determining, by the computingapparatus, a total number of communication packets to request for eachrespective network path, based on the quality predicted for therespective network path; requesting, by the computing apparatus, aplurality of first communication packets to be received via theplurality of network paths, wherein the plurality of first communicationpackets includes (i) systematic packets that each include unmodifiedoriginal information from a respective original data packet and (ii)coded packets that each include modified original information from atleast one of a plurality of the original data packets; and receiving, bythe computing apparatus, at least some of the first communicationpackets in parallel via the plurality of network paths, wherein thefirst communication packets are distributed for transmission among theplurality of network paths based on the total number determined for eachrespective network path.
 2. The method of claim 1, wherein each of theplurality of network paths is associated with a single server devicestoring the data segment.
 3. The method of claim 1, further comprisingdetermining, by the computing apparatus, a number of systematic packetsto request for each of the plurality of network paths, wherein thesystematic packets are distributed for transmission among the networkpaths based on the respective number of systematic packets determinedfor each network path.
 4. The method of claim 3, wherein the codedpackets are transmitted subsequent to any of the systematic packetstransmitted on the same network path.
 5. The method of claim 4, whereineach coded packet is generated to be linearly independent of at leastone selected from a group consisting of (i) one or more of thesystematic packets, (ii) any of the systematic packets that aretransmitted on the same network path before each coded packet istransmitted on its respective network path, (iii) any of the systematicpackets that are transmitted on any of the plurality of network pathsbefore each coded packet is transmitted on its respective network path,(iv) any of the coded packets that are transmitted on the same networkpath before each coded packet is transmitted on its respective networkpath, and (v) any of the coded packets that are transmitted on any ofthe plurality of network paths before each coded packet is transmittedon its respective network path.
 6. The method of claim 3, wherein thesystematic packets are distributed among the network pathsproportionally to the respective quality predicted for each networkpath.
 7. The method of claim 3, wherein the systematic packets aredistributed on a first subset of the plurality of network paths andwherein the coded packets are distributed on a second subset of theplurality of network paths different from the first subset of theplurality of network paths.
 8. The method of claim 1, whereindetermining the total number of communication packets to request foreach respective path comprises further optimizing by determining anoptimized number of communication packets to request for the respectivepath, and then increasing the optimized number to provide the totalnumber of communication packets to transmit.
 9. The method of claim 1,wherein the coded packets that each include the modified originalinformation from the respective original data packet are generated bymodifying original information from the respective original data packetusing a forward error correction technique.
 10. The method of claim 1,further comprising transmitting, by the computing apparatus, anindication to terminate transmission of generated first communicationpackets remaining to be transmitted, wherein the indication istransmitted by the computing apparatus in response to a determinationthat a combined set of received first communication packets issufficient to recover the data segment.
 11. The method of claim 1,further comprising: determining, by the computing apparatus, atransmission rate for each respective network path based on the qualitypredicted for the respective network path based on at least one ofbandwidth of each respective network path, latency of each respectivenetwork path, or packet loss of each respective network path.
 12. Themethod of claim 11, wherein the transmission rate for each respectivenetwork path is determined to be equal to the bandwidth of eachrespective network path.
 13. The method of claim 1, further comprisingdetermining a number of network paths to be used for receiving the firstcommunication packets.
 14. The method of claim 13, wherein the datasegment has a transmission time window, and wherein the number ofnetwork paths to be used is based on the transmission time window. 15.The method of claim 1, wherein the data segment has a transmission timewindow, and wherein determining the total number of communicationpackets to request for each respective network path is further based onthe transmission time window.
 16. The method of claim 1, wherein thedata segment has an overall delivery time, and determining the totalnumber of communication packets to request for each respective networkpath further comprises optimizing the total number based on aminimization of the overall delivery time.
 17. The method of claim 1,wherein determining the total number of communication packets to requestfor each respective network path is further based on a decoding successfor each of the plurality of network paths.
 18. The method of claim 1,wherein determining the total number of communication packets to requestfor each respective network path is further based on optimizationwithout a limited transmission time window for the data segment.
 19. Themethod of claim 1, wherein determining the total number of communicationpackets to request for each respective network path further comprisesadjusting a number of communication packets for the respective networkpath where a limited transmission time window limits the respectivenetwork path, wherein adjusting the number of communication packetscomprises relocating leftover packets to other network paths that areable to transmit an additional number of communication packets.
 20. Themethod of claim 19, further comprising calculating an overall deliverytime for the data segment based on the adjusting of the number ofcommunication packets.
 21. A non-transitory computer-readable storagemedium storing instructions which, when executed by a computingapparatus, cause the computing apparatus to perform a method, the methodcomprising: predicting, by the computing apparatus, a quality for eachof a plurality of network paths between the computing apparatus and oneor more server devices storing a data segment comprising a number oforiginal data packets; determining, by the computing apparatus, a totalnumber of communication packets to request for each respective networkpath, based on the quality predicted for the respective network path;requesting, by the computing apparatus, a plurality of firstcommunication packets to be received via the plurality of network paths,wherein the plurality of first communication packets includes (i)systematic packets that each include unmodified original informationfrom a respective original data packet and (ii) coded packets that eachinclude modified original information from at least one of a pluralityof the original data packets; and receiving, by the computing apparatus,at least some of the first communication packets in parallel via theplurality of network paths, wherein the first communication packets aredistributed for transmission among the plurality of network paths basedon the total number determined for each respective network path.
 22. Acomputing apparatus, comprising: at least one processor; and memorystoring instructions, which when executed by the at least one processor,cause the computing apparatus to perform the steps of: predicting aquality for each of a plurality of network paths between the computingapparatus and one or more server devices storing a data segmentcomprising a number of original data packets; determining a total numberof communication packets to request for each respective network path,based on the quality predicted for the respective network path;requesting a plurality of first communication packets to be received viathe plurality of network paths, wherein the plurality of firstcommunication packets includes (i) systematic packets that each includeunmodified original information from a respective original data packetand (ii) coded packets that each include modified original informationfrom at least one of a plurality of the original data packets and thatare linearly independent of one or more systematic packets; andreceiving at least some of the first communication packets in parallelvia the plurality of network paths, wherein the first communicationpackets are distributed for transmission among the plurality of networkpaths based on the total number determined for each respective networkpath.